[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:49:25 dist-test-slave-p3ml krb5kdc[971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:49:25 dist-test-slave-p3ml krb5kdc[971](info): set up 2 sockets
Jun 19 14:49:25 dist-test-slave-p3ml krb5kdc[971](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 14:49:26.413017 964 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:49:26 dist-test-slave-p3ml krb5kdc[971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:49:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:49:26.582731 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33397
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:33397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:26.897832 984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:26.898102 984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:26.898219 984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:26.905257 984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:26.905364 984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:26.905488 984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:26.905591 984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:26.905678 984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:26.915745 984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:33397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:26.917533 984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:26.918823 984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:26.930742 989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:26.930691 992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:26.930666 990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:26.931803 984 server_base.cc:1034] running on GCE node
I20260619 14:49:26.932571 984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:26.933866 984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:26.935103 984 hybrid_clock.cc:648] HybridClock initialized: now 1781880566935033 us; error 49 us; skew 500 ppm
I20260619 14:49:26.935541 984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:26 dist-test-slave-p3ml krb5kdc[971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:26.938973 984 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:49:26.940518 984 webserver.cc:460] Webserver started at http://127.0.241.62:40475/ using document root <none> and password file <none>
I20260619 14:49:26.941205 984 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:26.941354 984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:26.941666 984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:26.944630 984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "ecd803005f124e208be19b658bf35f8c"
format_stamp: "Formatted at 2026-06-19 14:49:26 on dist-test-slave-p3ml"
server_key: "d02582b1605b3e5d918eb9b162ac8a8d"
server_key_iv: "e26e463e97d7d73935ea8ae6c0907d7d"
server_key_version: "encryptionkey@0"
I20260619 14:49:26.945382 984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "ecd803005f124e208be19b658bf35f8c"
format_stamp: "Formatted at 2026-06-19 14:49:26 on dist-test-slave-p3ml"
server_key: "d02582b1605b3e5d918eb9b162ac8a8d"
server_key_iv: "e26e463e97d7d73935ea8ae6c0907d7d"
server_key_version: "encryptionkey@0"
I20260619 14:49:26.950263 984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 14:49:26.953604 999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:26.955153 984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:49:26.955340 984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "ecd803005f124e208be19b658bf35f8c"
format_stamp: "Formatted at 2026-06-19 14:49:26 on dist-test-slave-p3ml"
server_key: "d02582b1605b3e5d918eb9b162ac8a8d"
server_key_iv: "e26e463e97d7d73935ea8ae6c0907d7d"
server_key_version: "encryptionkey@0"
I20260619 14:49:26.955611 984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:26.978147 984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:26.981925 984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:26.982208 984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:27.001812 984 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:33397
I20260619 14:49:27.001857 1050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:33397 every 8 connection(s)
I20260619 14:49:27.003230 984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:27.006822 1051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:27.008967 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 984
I20260619 14:49:27.009284 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:49:27.011722 964 external_mini_cluster.cc:1428] Setting key fa0fa89b4a711477bba4939b4886a0a7
I20260619 14:49:27.019694 1051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c: Bootstrap starting.
I20260619 14:49:27.022485 1051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:27.023473 1051 log.cc:824] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:27.025941 1051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c: No bootstrap required, opened a new log
I20260619 14:49:27.035486 1051 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd803005f124e208be19b658bf35f8c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33397 } }
I20260619 14:49:27.035900 1051 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:27.036019 1051 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd803005f124e208be19b658bf35f8c, State: Initialized, Role: FOLLOWER
I20260619 14:49:27.036628 1051 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd803005f124e208be19b658bf35f8c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33397 } }
I20260619 14:49:27.036890 1051 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:27.037026 1051 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:27.037206 1051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:27.040292 1051 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd803005f124e208be19b658bf35f8c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33397 } }
I20260619 14:49:27.040771 1051 leader_election.cc:302] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecd803005f124e208be19b658bf35f8c; no voters:
I20260619 14:49:27.041350 1051 leader_election.cc:288] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:49:27.041527 1056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:27.043334 1051 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:27.045599 1056 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [term 1 LEADER]: Becoming Leader. State: Replica: ecd803005f124e208be19b658bf35f8c, State: Running, Role: LEADER
Jun 19 14:49:27 dist-test-slave-p3ml krb5kdc[971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:27.046103 1056 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd803005f124e208be19b658bf35f8c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33397 } }
I20260619 14:49:27.054003 1057 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecd803005f124e208be19b658bf35f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd803005f124e208be19b658bf35f8c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33397 } } }
I20260619 14:49:27.054005 1058 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecd803005f124e208be19b658bf35f8c. Latest consensus state: current_term: 1 leader_uuid: "ecd803005f124e208be19b658bf35f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd803005f124e208be19b658bf35f8c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33397 } } }
I20260619 14:49:27.054479 1057 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:27.054486 1058 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:27.055366 1071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:27.064854 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:27.015964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.0.241.62:33397)
0619 14:49:27.017365 (+ 1401us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:27.017380 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:27.020929 (+ 3549us) server_negotiation.cc:407] Connection header received
0619 14:49:27.031034 (+ 10105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:27.031058 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:27.031584 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:27.032025 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:27.034610 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:27.036532 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:27.038351 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:27.039791 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:27.039967 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:27.048014 (+ 8047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:27.048029 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:27.048068 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:27.048145 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:27.054352 (+ 6207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:27.055836 (+ 1484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:27.055848 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:27.055860 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:27.056139 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:27.056875 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:27.056883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:27.056889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:27.057799 (+ 910us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:27.058178 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:27.062110 (+ 3932us) server_negotiation.cc:299] Negotiation successful
0619 14:49:27.062646 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":915,"threads_started":1}
I20260619 14:49:27.067988 1071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:27.076484 1071 catalog_manager.cc:1347] Generated new cluster ID: 12c33290c0a44a89801936e8396af431
I20260619 14:49:27.076612 1071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:27.087414 1071 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:27.088629 1071 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:27.095048 1071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecd803005f124e208be19b658bf35f8c: Generated new TSK 0
I20260619 14:49:27.095870 1071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:49:27.139146 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33397
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:27.468672 1077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:27.468963 1077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:27.469089 1077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:27.476761 1077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:27.476869 1077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:27.477077 1077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:49:27.486738 1077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:27.488040 1077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:27.489307 1077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:27.502116 1083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:27.502152 1085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:27.502784 1082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:27.503571 1077 server_base.cc:1034] running on GCE node
I20260619 14:49:27.504123 1077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:27.504961 1077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:27.506202 1077 hybrid_clock.cc:648] HybridClock initialized: now 1781880567506164 us; error 32 us; skew 500 ppm
I20260619 14:49:27.506501 1077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:27 dist-test-slave-p3ml krb5kdc[971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:27.509558 1077 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:49:27.510850 1077 webserver.cc:460] Webserver started at http://127.0.241.1:41097/ using document root <none> and password file <none>
I20260619 14:49:27.511547 1077 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:27.511647 1077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:27.512297 1077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:27.515002 1077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "1c4a3d88eafc45da87c25d17ee268daa"
format_stamp: "Formatted at 2026-06-19 14:49:27 on dist-test-slave-p3ml"
server_key: "c7a5dda5058159a5c9fdac0e0e3afa91"
server_key_iv: "24d3bd3513bcc367c7ac1cbacf305ccf"
server_key_version: "encryptionkey@0"
I20260619 14:49:27.515785 1077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "1c4a3d88eafc45da87c25d17ee268daa"
format_stamp: "Formatted at 2026-06-19 14:49:27 on dist-test-slave-p3ml"
server_key: "c7a5dda5058159a5c9fdac0e0e3afa91"
server_key_iv: "24d3bd3513bcc367c7ac1cbacf305ccf"
server_key_version: "encryptionkey@0"
I20260619 14:49:27.520774 1077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 14:49:27.524036 1092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:27.525413 1077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:49:27.525569 1077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "1c4a3d88eafc45da87c25d17ee268daa"
format_stamp: "Formatted at 2026-06-19 14:49:27 on dist-test-slave-p3ml"
server_key: "c7a5dda5058159a5c9fdac0e0e3afa91"
server_key_iv: "24d3bd3513bcc367c7ac1cbacf305ccf"
server_key_version: "encryptionkey@0"
I20260619 14:49:27.525749 1077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:27.540973 1077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:27.546168 1077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:27.546411 1077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:27.547430 1077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:27.548808 1077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:27.548880 1077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:27.549034 1077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:27.549096 1077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:27.585001 1077 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:39257
I20260619 14:49:27.585033 1204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:39257 every 8 connection(s)
I20260619 14:49:27.586140 1077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:27.595798 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 1077
I20260619 14:49:27.596016 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:49:27.596621 964 external_mini_cluster.cc:1428] Setting key ed8ff78f2fab738fe3d786242410d0bb
Jun 19 14:49:27 dist-test-slave-p3ml krb5kdc[971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:27.605862 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:27.588706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:59557 (local address 127.0.241.62:33397)
0619 14:49:27.589173 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:27.589179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:27.590084 (+ 905us) server_negotiation.cc:407] Connection header received
0619 14:49:27.591859 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:27.591864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:27.591935 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:27.592030 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:27.593427 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:27.594126 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:27.595826 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:27.596311 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:27.596443 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:27.600554 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:27.600558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:27.600562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:27.600589 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:27.602979 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:27.603714 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:27.603720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:27.603728 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:27.603833 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:27.604248 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:27.604253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:27.604256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:27.604499 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:27.604645 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:27.605303 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 14:49:27.605519 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":219,"threads_started":1}
I20260619 14:49:27.606757 1208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:27.589238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33397 (local address 127.0.241.1:59557)
0619 14:49:27.589923 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:27.589960 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:27.591622 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:27.592234 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:27.592242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:27.592730 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:27.593286 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:27.593302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:27.594313 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:27.594318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:27.595682 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:27.595694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:27.597022 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:27.597031 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:27.597231 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:27.597992 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:27.598025 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:27.600337 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:27.603107 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:27.603115 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:27.603130 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:27.603597 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:27.603935 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:27.603941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:27.603945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:27.604132 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:27.604619 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:27.604627 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:27.604936 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 14:49:27.605211 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 14:49:27.605699 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":269,"threads_started":1}
I20260619 14:49:27.608266 1205 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33397
I20260619 14:49:27.608561 1205 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:27.609258 1205 heartbeater.cc:511] Master 127.0.241.62:33397 requested a full tablet report, sending...
I20260619 14:49:27.611233 1016 ts_manager.cc:194] Registered new tserver with Master: 1c4a3d88eafc45da87c25d17ee268daa (127.0.241.1:39257)
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
I20260619 14:49:27.612637 1016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:59557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:49:27.634126 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33397
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:27.955325 1211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:27.955634 1211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:27.955823 1211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:27.963598 1211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:27.963701 1211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:27.963918 1211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:49:27.973315 1211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:27.974553 1211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:27.975898 1211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:27.990195 1219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:27.990231 1217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:27.990267 1216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:27.991250 1211 server_base.cc:1034] running on GCE node
I20260619 14:49:27.991874 1211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:27.992739 1211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:27.993969 1211 hybrid_clock.cc:648] HybridClock initialized: now 1781880567993937 us; error 45 us; skew 500 ppm
I20260619 14:49:27.994294 1211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:27 dist-test-slave-p3ml krb5kdc[971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:28.000088 1211 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:49:28.001905 1211 webserver.cc:460] Webserver started at http://127.0.241.2:41191/ using document root <none> and password file <none>
I20260619 14:49:28.002725 1211 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:28.002859 1211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:28.003185 1211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:28.006816 1211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "05db01c6d407483bbef1273bfeb5c4f5"
format_stamp: "Formatted at 2026-06-19 14:49:28 on dist-test-slave-p3ml"
server_key: "d85db59dcfca761caadfb695618ef1ba"
server_key_iv: "5ac36162503bc54c8861bd6502057b0e"
server_key_version: "encryptionkey@0"
I20260619 14:49:28.008201 1211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "05db01c6d407483bbef1273bfeb5c4f5"
format_stamp: "Formatted at 2026-06-19 14:49:28 on dist-test-slave-p3ml"
server_key: "d85db59dcfca761caadfb695618ef1ba"
server_key_iv: "5ac36162503bc54c8861bd6502057b0e"
server_key_version: "encryptionkey@0"
I20260619 14:49:28.014376 1211 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 14:49:28.018460 1226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:28.019892 1211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:28.020071 1211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "05db01c6d407483bbef1273bfeb5c4f5"
format_stamp: "Formatted at 2026-06-19 14:49:28 on dist-test-slave-p3ml"
server_key: "d85db59dcfca761caadfb695618ef1ba"
server_key_iv: "5ac36162503bc54c8861bd6502057b0e"
server_key_version: "encryptionkey@0"
I20260619 14:49:28.020238 1211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:28.042467 1211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:28.047040 1211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:28.047330 1211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:28.048322 1211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:28.049790 1211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:28.049894 1211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:28.050010 1211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:28.050058 1211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:28.083360 1211 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:38711
I20260619 14:49:28.083457 1338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:38711 every 8 connection(s)
I20260619 14:49:28.084991 1211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:49:28.090276 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 1211
I20260619 14:49:28.090471 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:49:28.090880 964 external_mini_cluster.cc:1428] Setting key f2779fb7e5e05c3680f59cbf4ba4db90
Jun 19 14:49:28 dist-test-slave-p3ml krb5kdc[971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:28.107004 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:28.087839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:60841 (local address 127.0.241.62:33397)
0619 14:49:28.088014 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:28.088019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:28.089531 (+ 1512us) server_negotiation.cc:407] Connection header received
0619 14:49:28.091546 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:28.091552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:28.091637 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:28.091756 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:28.093163 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.093882 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.095536 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.096274 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.096416 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:28.100230 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:28.100239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:28.100247 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:28.100290 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:28.103843 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.104568 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.104576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.104582 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.104747 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.105158 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.105165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.105168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.105531 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:28.105677 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:28.106312 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 14:49:28.106559 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
I20260619 14:49:28.107765 1341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:28.088276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33397 (local address 127.0.241.2:60841)
0619 14:49:28.089365 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:28.089407 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:28.091246 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:28.091983 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:28.091991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:28.092470 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:28.092986 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:28.093006 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.094021 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.094026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:28.095352 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:28.095369 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.096503 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.096513 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:28.096826 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:28.097571 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:28.097608 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:28.099998 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:28.104011 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:28.104017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:28.104028 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:28.104429 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:28.104860 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:28.104863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:28.104866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:28.105026 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:28.105648 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:28.105653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:28.105936 (+ 283us) client_negotiation.cc:769] Sending connection context
0619 14:49:28.106164 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 14:49:28.106509 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":273,"threads_started":1}
I20260619 14:49:28.109198 1339 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33397
I20260619 14:49:28.109508 1339 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
I20260619 14:49:28.110685 1339 heartbeater.cc:511] Master 127.0.241.62:33397 requested a full tablet report, sending...
I20260619 14:49:28.112159 1016 ts_manager.cc:194] Registered new tserver with Master: 05db01c6d407483bbef1273bfeb5c4f5 (127.0.241.2:38711)
I20260619 14:49:28.112919 1016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:60841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:49:28.130254 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33397
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:28.469489 1344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:28.469765 1344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:28.469951 1344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:28.477291 1344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:28.477413 1344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:28.477600 1344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:49:28.487865 1344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33397
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:28.489157 1344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:28.490540 1344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:28.503782 1349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:28.503786 1350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:28.504060 1352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:28.505106 1344 server_base.cc:1034] running on GCE node
I20260619 14:49:28.505713 1344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:28.506676 1344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:28.507898 1344 hybrid_clock.cc:648] HybridClock initialized: now 1781880568507860 us; error 30 us; skew 500 ppm
I20260619 14:49:28.508229 1344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:28 dist-test-slave-p3ml krb5kdc[971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:28.511366 1344 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:49:28.512794 1344 webserver.cc:460] Webserver started at http://127.0.241.3:38049/ using document root <none> and password file <none>
I20260619 14:49:28.513470 1344 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:28.513614 1344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:28.513914 1344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:28.516805 1344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "1ffbfea8283d4d87be81909b97b1cd08"
format_stamp: "Formatted at 2026-06-19 14:49:28 on dist-test-slave-p3ml"
server_key: "59dcb9b16dc1549407419e9822751b0e"
server_key_iv: "18432f33dd81c4fe6e88c205642dbfcc"
server_key_version: "encryptionkey@0"
I20260619 14:49:28.517580 1344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "1ffbfea8283d4d87be81909b97b1cd08"
format_stamp: "Formatted at 2026-06-19 14:49:28 on dist-test-slave-p3ml"
server_key: "59dcb9b16dc1549407419e9822751b0e"
server_key_iv: "18432f33dd81c4fe6e88c205642dbfcc"
server_key_version: "encryptionkey@0"
I20260619 14:49:28.522565 1344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 14:49:28.525988 1359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:28.527153 1344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:49:28.527304 1344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "1ffbfea8283d4d87be81909b97b1cd08"
format_stamp: "Formatted at 2026-06-19 14:49:28 on dist-test-slave-p3ml"
server_key: "59dcb9b16dc1549407419e9822751b0e"
server_key_iv: "18432f33dd81c4fe6e88c205642dbfcc"
server_key_version: "encryptionkey@0"
I20260619 14:49:28.527531 1344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:28.545730 1344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:28.550166 1344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:28.550501 1344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:28.551497 1344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:28.552789 1344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:28.552867 1344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:28.553045 1344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:28.553112 1344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:28.586283 1344 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:32781
I20260619 14:49:28.586376 1471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:32781 every 8 connection(s)
I20260619 14:49:28.587668 1344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:49:28.597061 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 1344
I20260619 14:49:28.597271 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:49:28.597714 964 external_mini_cluster.cc:1428] Setting key 73f6939b47eb7ebe2d6bb4b2085f3124
Jun 19 14:49:28 dist-test-slave-p3ml krb5kdc[971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:28.607498 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:28.590310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:46057 (local address 127.0.241.62:33397)
0619 14:49:28.590518 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:28.590525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:28.591675 (+ 1150us) server_negotiation.cc:407] Connection header received
0619 14:49:28.593344 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:28.593348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:28.593444 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:28.593533 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:28.594896 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.595730 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.597249 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.597736 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.597873 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:28.601717 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:28.601722 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:28.601725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:28.601755 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:28.604598 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.605294 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.605301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.605305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.605437 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.605854 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.605861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.605865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.606134 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:28.606291 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:28.606919 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 14:49:28.607135 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:49:28.608399 1474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:28.590806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33397 (local address 127.0.241.3:46057)
0619 14:49:28.591531 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:28.591569 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:28.593106 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:28.593723 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:28.593731 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:28.594213 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:28.594738 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:28.594754 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.595859 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.595863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:28.597091 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:28.597103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.597968 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.597974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:28.598219 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:28.598894 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:28.598923 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:28.601503 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:28.604738 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:28.604746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:28.604762 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:28.605173 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:28.605554 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:28.605558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:28.605562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:28.605743 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:28.606266 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:28.606274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:28.606586 (+ 312us) client_negotiation.cc:769] Sending connection context
0619 14:49:28.606842 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 14:49:28.607337 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":267,"threads_started":1}
I20260619 14:49:28.609813 1472 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33397
I20260619 14:49:28.610155 1472 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:28.610901 1472 heartbeater.cc:511] Master 127.0.241.62:33397 requested a full tablet report, sending...
I20260619 14:49:28.612445 1016 ts_manager.cc:194] Registered new tserver with Master: 1ffbfea8283d4d87be81909b97b1cd08 (127.0.241.3:32781)
I20260619 14:49:28.613454 1016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:46057
I20260619 14:49:28.613653 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:49:28.615640 1205 heartbeater.cc:503] Master 127.0.241.62:33397 was elected leader, sending a full tablet report...
Jun 19 14:49:28 dist-test-slave-p3ml krb5kdc[971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880568, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:49:28 dist-test-slave-p3ml krb5kdc[971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880568, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:49:28.657423 1478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:28.642403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.0.241.1:39257)
0619 14:49:28.642997 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:28.643003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:28.643121 (+ 118us) server_negotiation.cc:407] Connection header received
0619 14:49:28.643260 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:28.643266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:28.643573 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:28.643705 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:28.645096 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.646074 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.647243 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.647853 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.647948 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:28.652233 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:28.652244 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:28.652259 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:28.652295 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:28.654972 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.655619 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.655625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.655628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.655765 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.656096 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.656099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.656101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.656413 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:28.656638 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:28.656850 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 14:49:28.657035 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":296,"threads_started":1}
W20260619 14:49:28.658630 1120 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39568
Jun 19 14:49:28 dist-test-slave-p3ml krb5kdc[971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:49:28 dist-test-slave-p3ml krb5kdc[971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:49:28.701442 1478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:28.687961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.0.241.1:39257)
0619 14:49:28.688205 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:28.688211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:28.688621 (+ 410us) server_negotiation.cc:407] Connection header received
0619 14:49:28.688833 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:28.688865 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:28.688973 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:28.689096 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:28.690392 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.691098 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.692364 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.692777 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.692927 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:28.696808 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:28.696813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:28.696816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:28.696845 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:28.699408 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.700008 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.700020 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.700022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.700134 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.700431 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.700434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.700436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.700653 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:28.700820 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:28.701064 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 14:49:28.701216 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 14:49:28.702828 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 1077
I20260619 14:49:28.714097 1478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:28.703762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.0.241.1:39257)
0619 14:49:28.703923 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:28.703927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:28.704234 (+ 307us) server_negotiation.cc:407] Connection header received
0619 14:49:28.704327 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:28.704330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:28.704379 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:28.704451 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:28.705473 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.706117 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.707242 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.707724 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.707815 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:28.709762 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:28.709767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:28.709770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:28.709795 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:28.712181 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.712709 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.712712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.712714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.712823 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.713121 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.713124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.713126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.713333 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:28.713490 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:28.713722 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 14:49:28.713851 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:49:28.714695 1200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:28.796046 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 1077
I20260619 14:49:28.811813 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 1211
Jun 19 14:49:28 dist-test-slave-p3ml krb5kdc[971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:49:28.827148 1486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:28.812602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.0.241.2:38711)
0619 14:49:28.813070 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:28.813077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:28.813099 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:49:28.813160 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:28.813166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:28.813387 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:28.813566 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:28.814747 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.815804 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.817012 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.817528 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.817667 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:28.821627 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:28.821636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:28.821650 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:28.821681 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:28.824389 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.825057 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.825063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.825066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.825197 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.825577 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.825582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.825587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.825905 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:28.826217 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:28.826549 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 14:49:28.826807 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":237,"threads_started":1}
I20260619 14:49:28.828601 1334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:28.909920 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 1211
I20260619 14:49:28.931617 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 1344
Jun 19 14:49:28 dist-test-slave-p3ml krb5kdc[971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:49:28.947587 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:28.932657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.0.241.3:32781)
0619 14:49:28.933220 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:28.933227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:28.933253 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:49:28.933310 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:28.933316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:28.933518 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:28.933685 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:28.934903 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.935994 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.937260 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:28.937820 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:28.937953 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:28.941893 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:28.941902 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:28.941923 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:28.941959 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:28.944861 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.945482 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.945490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.945493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.945606 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:28.945961 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:28.945968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:28.945971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:28.946308 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:28.946613 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:28.946900 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:49:28.947174 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":279,"threads_started":1}
I20260619 14:49:28.948808 1467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:29.030169 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 1344
I20260619 14:49:29.046133 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 984
I20260619 14:49:29.047276 1046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:29.148274 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 984
2026-06-19T14:49:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3832 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:49:29 dist-test-slave-p3ml krb5kdc[1493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:49:29 dist-test-slave-p3ml krb5kdc[1493](info): set up 2 sockets
Jun 19 14:49:29 dist-test-slave-p3ml krb5kdc[1493](info): commencing operation
krb5kdc: starting...
W20260619 14:49:30.235370 964 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:49:30 dist-test-slave-p3ml krb5kdc[1493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:49:30.404868 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:37097
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:37097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:30.718537 1506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:30.718879 1506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:30.719079 1506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:30.726339 1506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:30.726490 1506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:30.726598 1506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:30.726701 1506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:30.726776 1506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:30.736593 1506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:37097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:30.737907 1506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:30.739257 1506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:30.751197 1512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:30.751243 1511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:30.751420 1514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:30.752084 1506 server_base.cc:1034] running on GCE node
I20260619 14:49:30.752812 1506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:30.754130 1506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:30.755357 1506 hybrid_clock.cc:648] HybridClock initialized: now 1781880570755326 us; error 47 us; skew 500 ppm
I20260619 14:49:30.755717 1506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:30 dist-test-slave-p3ml krb5kdc[1493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:30.758651 1506 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:49:30.760208 1506 webserver.cc:460] Webserver started at http://127.0.241.62:39413/ using document root <none> and password file <none>
I20260619 14:49:30.760885 1506 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:30.761026 1506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:30.761339 1506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:30.764312 1506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "8a77b7ae9dd141edbc677fcd5eb5de7e"
format_stamp: "Formatted at 2026-06-19 14:49:30 on dist-test-slave-p3ml"
server_key: "9da1c05baf1497c822b876b99819173a"
server_key_iv: "74cbd4008d4628b510369e0639d28fc3"
server_key_version: "encryptionkey@0"
I20260619 14:49:30.765056 1506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "8a77b7ae9dd141edbc677fcd5eb5de7e"
format_stamp: "Formatted at 2026-06-19 14:49:30 on dist-test-slave-p3ml"
server_key: "9da1c05baf1497c822b876b99819173a"
server_key_iv: "74cbd4008d4628b510369e0639d28fc3"
server_key_version: "encryptionkey@0"
I20260619 14:49:30.770126 1506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:49:30.773331 1521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:30.774559 1506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:49:30.774740 1506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "8a77b7ae9dd141edbc677fcd5eb5de7e"
format_stamp: "Formatted at 2026-06-19 14:49:30 on dist-test-slave-p3ml"
server_key: "9da1c05baf1497c822b876b99819173a"
server_key_iv: "74cbd4008d4628b510369e0639d28fc3"
server_key_version: "encryptionkey@0"
I20260619 14:49:30.774917 1506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:30.793308 1506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:30.796911 1506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:30.797180 1506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:30.814561 1506 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:37097
I20260619 14:49:30.814597 1572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:37097 every 8 connection(s)
I20260619 14:49:30.815964 1506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:30.819491 1573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:30.821033 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 1506
I20260619 14:49:30.821244 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:49:30.821854 964 external_mini_cluster.cc:1428] Setting key b78bea71853ebde208925c93b2333d10
I20260619 14:49:30.833086 1573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e: Bootstrap starting.
I20260619 14:49:30.836658 1573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:30.837625 1573 log.cc:824] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:49:30 dist-test-slave-p3ml krb5kdc[1493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:30.840798 1573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e: No bootstrap required, opened a new log
I20260619 14:49:30.848053 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.823806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.0.241.62:37097)
0619 14:49:30.826736 (+ 2930us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:30.826753 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:30.826801 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:49:30.828900 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:30.828928 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:30.829454 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:30.829891 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:30.831150 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.833069 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.834363 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.835419 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.835622 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:30.839394 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:30.839404 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:30.839421 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:30.839493 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:30.843335 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:30.844066 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:30.844075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:30.844083 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:30.844274 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:30.844615 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:30.844620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:30.844624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:30.845341 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:30.845631 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:30.846054 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 14:49:30.846535 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2704,"thread_start_us":2491,"threads_started":1}
I20260619 14:49:30.851274 1573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a77b7ae9dd141edbc677fcd5eb5de7e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37097 } }
I20260619 14:49:30.851696 1573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:30.851821 1573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a77b7ae9dd141edbc677fcd5eb5de7e, State: Initialized, Role: FOLLOWER
I20260619 14:49:30.852378 1573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a77b7ae9dd141edbc677fcd5eb5de7e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37097 } }
I20260619 14:49:30.852625 1573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:30.852751 1573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:30.852923 1573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:30.855233 1573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a77b7ae9dd141edbc677fcd5eb5de7e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37097 } }
I20260619 14:49:30.855782 1573 leader_election.cc:302] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a77b7ae9dd141edbc677fcd5eb5de7e; no voters:
I20260619 14:49:30.856355 1573 leader_election.cc:288] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:49:30.856510 1578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:30.857393 1578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [term 1 LEADER]: Becoming Leader. State: Replica: 8a77b7ae9dd141edbc677fcd5eb5de7e, State: Running, Role: LEADER
I20260619 14:49:30.857939 1578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a77b7ae9dd141edbc677fcd5eb5de7e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37097 } }
I20260619 14:49:30.858527 1573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:30.861196 1579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a77b7ae9dd141edbc677fcd5eb5de7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a77b7ae9dd141edbc677fcd5eb5de7e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37097 } } }
I20260619 14:49:30.861253 1580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a77b7ae9dd141edbc677fcd5eb5de7e. Latest consensus state: current_term: 1 leader_uuid: "8a77b7ae9dd141edbc677fcd5eb5de7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a77b7ae9dd141edbc677fcd5eb5de7e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37097 } } }
I20260619 14:49:30.861660 1579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:30.861712 1580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:30.863130 1587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:30.867419 1587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:30.876121 1587 catalog_manager.cc:1347] Generated new cluster ID: 9ab14e45576e4fbe939c90cc658e8200
I20260619 14:49:30.876245 1587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:30.895774 1587 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:30.897635 1587 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:30.908982 1587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a77b7ae9dd141edbc677fcd5eb5de7e: Generated new TSK 0
I20260619 14:49:30.910051 1587 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:49:30.962642 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:37097
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:31.281467 1599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:31.281765 1599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:31.281929 1599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:31.289265 1599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:31.289397 1599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:31.289603 1599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:49:31.299268 1599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:37097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:31.300591 1599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:31.301889 1599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:31.314456 1604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:31.314415 1607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:31.314534 1605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:31.315732 1599 server_base.cc:1034] running on GCE node
I20260619 14:49:31.316321 1599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:31.317133 1599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:31.318323 1599 hybrid_clock.cc:648] HybridClock initialized: now 1781880571318277 us; error 40 us; skew 500 ppm
I20260619 14:49:31.318688 1599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:31 dist-test-slave-p3ml krb5kdc[1493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:31.321702 1599 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:49:31.323052 1599 webserver.cc:460] Webserver started at http://127.0.241.1:40609/ using document root <none> and password file <none>
I20260619 14:49:31.323745 1599 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:31.323855 1599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:31.324085 1599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:31.326855 1599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "b8a20593f419418da9e3ee1058149f5d"
format_stamp: "Formatted at 2026-06-19 14:49:31 on dist-test-slave-p3ml"
server_key: "b6866124d69aad45d54e61320860aa97"
server_key_iv: "ca0b314a7b318984ce518eaa64cf8d2e"
server_key_version: "encryptionkey@0"
I20260619 14:49:31.327633 1599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "b8a20593f419418da9e3ee1058149f5d"
format_stamp: "Formatted at 2026-06-19 14:49:31 on dist-test-slave-p3ml"
server_key: "b6866124d69aad45d54e61320860aa97"
server_key_iv: "ca0b314a7b318984ce518eaa64cf8d2e"
server_key_version: "encryptionkey@0"
I20260619 14:49:31.332460 1599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:49:31.335608 1614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:31.336848 1599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:49:31.337021 1599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "b8a20593f419418da9e3ee1058149f5d"
format_stamp: "Formatted at 2026-06-19 14:49:31 on dist-test-slave-p3ml"
server_key: "b6866124d69aad45d54e61320860aa97"
server_key_iv: "ca0b314a7b318984ce518eaa64cf8d2e"
server_key_version: "encryptionkey@0"
I20260619 14:49:31.337210 1599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:31.354722 1599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:31.358458 1599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:31.358779 1599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:31.359771 1599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:31.361055 1599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:31.361123 1599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:31.361286 1599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:31.361366 1599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:31.396008 1599 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:45243
I20260619 14:49:31.396055 1726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:45243 every 8 connection(s)
I20260619 14:49:31.397235 1599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:31.399361 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 1599
I20260619 14:49:31.399585 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:49:31.400318 964 external_mini_cluster.cc:1428] Setting key 9cac4b0efcb0876fff644b18224a80bd
Jun 19 14:49:31 dist-test-slave-p3ml krb5kdc[1493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
I20260619 14:49:31.418550 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.399729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:49315 (local address 127.0.241.62:37097)
0619 14:49:31.400134 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:31.400140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:31.401357 (+ 1217us) server_negotiation.cc:407] Connection header received
0619 14:49:31.403512 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:31.403517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:31.403597 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:31.403715 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:31.405132 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.405839 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.407628 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.408164 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.408267 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.412830 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:31.412834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:31.412837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:31.412865 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:31.415481 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:31.416378 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:31.416382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:31.416384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:31.416499 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:31.416972 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:31.416975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:31.416977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:31.417216 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:31.417317 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:31.418098 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 14:49:31.418249 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":210,"threads_started":1}
I20260619 14:49:31.419682 1730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.400350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37097 (local address 127.0.241.1:49315)
0619 14:49:31.401194 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:31.401246 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:31.403213 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:31.403948 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:31.403956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:31.404439 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:31.404970 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.404990 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.405990 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.405996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.407489 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.407507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.408391 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.408400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.408726 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.409460 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:31.409504 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:31.412623 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:31.415639 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:31.415648 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:31.415667 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:31.416265 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:31.416607 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:31.416612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:31.416617 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:31.416858 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:31.417342 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:31.417350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:31.417791 (+ 441us) client_negotiation.cc:769] Sending connection context
0619 14:49:31.418016 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 14:49:31.418364 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":298,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
I20260619 14:49:31.421236 1727 heartbeater.cc:347] Connected to a master server at 127.0.241.62:37097
I20260619 14:49:31.421541 1727 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:31.422221 1727 heartbeater.cc:511] Master 127.0.241.62:37097 requested a full tablet report, sending...
I20260619 14:49:31.424304 1538 ts_manager.cc:194] Registered new tserver with Master: b8a20593f419418da9e3ee1058149f5d (127.0.241.1:45243)
I20260619 14:49:31.426385 1538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:49315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:49:31.442459 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:37097
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:31.758100 1733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:31.758375 1733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:31.758538 1733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:31.765960 1733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:31.766098 1733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:31.766285 1733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:49:31.775758 1733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:37097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:31.777051 1733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:31.778446 1733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:31.791463 1739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:31.791688 1738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:31.791937 1741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:31.792611 1733 server_base.cc:1034] running on GCE node
I20260619 14:49:31.793108 1733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:31.793900 1733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:31.795084 1733 hybrid_clock.cc:648] HybridClock initialized: now 1781880571795026 us; error 49 us; skew 500 ppm
I20260619 14:49:31.795378 1733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:31 dist-test-slave-p3ml krb5kdc[1493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:31.798266 1733 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:49:31.799662 1733 webserver.cc:460] Webserver started at http://127.0.241.2:34507/ using document root <none> and password file <none>
I20260619 14:49:31.800345 1733 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:31.800487 1733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:31.800840 1733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:31.803655 1733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "564e7c1a1ee34b2a859f064a3827bf12"
format_stamp: "Formatted at 2026-06-19 14:49:31 on dist-test-slave-p3ml"
server_key: "83324272368e7186db6b68d800aa9cf3"
server_key_iv: "92eb601ec857acac369089911c02f88c"
server_key_version: "encryptionkey@0"
I20260619 14:49:31.804363 1733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "564e7c1a1ee34b2a859f064a3827bf12"
format_stamp: "Formatted at 2026-06-19 14:49:31 on dist-test-slave-p3ml"
server_key: "83324272368e7186db6b68d800aa9cf3"
server_key_iv: "92eb601ec857acac369089911c02f88c"
server_key_version: "encryptionkey@0"
I20260619 14:49:31.809178 1733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 14:49:31.812316 1748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:31.813560 1733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:49:31.813683 1733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "564e7c1a1ee34b2a859f064a3827bf12"
format_stamp: "Formatted at 2026-06-19 14:49:31 on dist-test-slave-p3ml"
server_key: "83324272368e7186db6b68d800aa9cf3"
server_key_iv: "92eb601ec857acac369089911c02f88c"
server_key_version: "encryptionkey@0"
I20260619 14:49:31.813831 1733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:31.832391 1733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:31.836162 1733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:31.836383 1733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:31.837237 1733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:31.838449 1733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:31.838518 1733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:31.838594 1733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:31.838635 1733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:31.871196 1733 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:46673
I20260619 14:49:31.871243 1860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:46673 every 8 connection(s)
I20260619 14:49:31.872450 1733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:49:31.879083 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 1733
I20260619 14:49:31.879280 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:49:31.879817 964 external_mini_cluster.cc:1428] Setting key a91868581ca45bacf14142f22a80b6d9
Jun 19 14:49:31 dist-test-slave-p3ml krb5kdc[1493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:31.895196 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.874951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:55563 (local address 127.0.241.62:37097)
0619 14:49:31.875119 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:31.875124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:31.876309 (+ 1185us) server_negotiation.cc:407] Connection header received
0619 14:49:31.878047 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:31.878053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:31.878146 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:31.878258 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:31.880239 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.881103 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.883320 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.883772 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.883866 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.888206 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:31.888215 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:31.888224 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:31.888266 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:31.891883 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:31.892595 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:31.892604 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:31.892610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:31.892777 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:31.893174 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:31.893181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:31.893187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:31.893567 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:31.893726 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:31.894469 (+ 743us) server_negotiation.cc:299] Negotiation successful
0619 14:49:31.894728 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:49:31.895876 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.875470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37097 (local address 127.0.241.2:55563)
0619 14:49:31.876151 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:31.876187 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:31.877781 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:31.878495 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:31.878503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:31.879242 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:31.880060 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.880084 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.881258 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.881263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.883156 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.883175 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.883997 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.884007 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.884343 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.885091 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:31.885128 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:31.887977 (+ 2849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:31.892016 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:31.892023 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:31.892036 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:31.892446 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:31.892882 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:31.892886 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:31.892888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:31.893052 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:31.893685 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:31.893690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:31.893998 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 14:49:31.894319 (+ 321us) client_negotiation.cc:240] Negotiation successful
0619 14:49:31.894718 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":266,"threads_started":1}
I20260619 14:49:31.897253 1861 heartbeater.cc:347] Connected to a master server at 127.0.241.62:37097
I20260619 14:49:31.897621 1861 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
I20260619 14:49:31.898530 1861 heartbeater.cc:511] Master 127.0.241.62:37097 requested a full tablet report, sending...
I20260619 14:49:31.900043 1538 ts_manager.cc:194] Registered new tserver with Master: 564e7c1a1ee34b2a859f064a3827bf12 (127.0.241.2:46673)
I20260619 14:49:31.900789 1538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:55563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:49:31.920904 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:37097
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:32.233963 1866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:32.234256 1866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:32.234422 1866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:32.241591 1866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:32.241732 1866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:32.241926 1866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:49:32.251292 1866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:37097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:32.252832 1866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:32.254097 1866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:32.266916 1871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:32.266868 1874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:32.266834 1872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:32.268115 1866 server_base.cc:1034] running on GCE node
I20260619 14:49:32.268741 1866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:32.269711 1866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:32.270946 1866 hybrid_clock.cc:648] HybridClock initialized: now 1781880572270895 us; error 45 us; skew 500 ppm
I20260619 14:49:32.271291 1866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:32 dist-test-slave-p3ml krb5kdc[1493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:32.274391 1866 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:49:32.275915 1866 webserver.cc:460] Webserver started at http://127.0.241.3:43085/ using document root <none> and password file <none>
I20260619 14:49:32.276615 1866 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:32.276753 1866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:32.277052 1866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:32.280093 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "4247e4b83d6349e28d87ea29b15604b2"
format_stamp: "Formatted at 2026-06-19 14:49:32 on dist-test-slave-p3ml"
server_key: "55d5ad46f572c6e6854b04e49f25598f"
server_key_iv: "f43142b97a25fe7397e7799880c89194"
server_key_version: "encryptionkey@0"
I20260619 14:49:32.280875 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "4247e4b83d6349e28d87ea29b15604b2"
format_stamp: "Formatted at 2026-06-19 14:49:32 on dist-test-slave-p3ml"
server_key: "55d5ad46f572c6e6854b04e49f25598f"
server_key_iv: "f43142b97a25fe7397e7799880c89194"
server_key_version: "encryptionkey@0"
I20260619 14:49:32.285856 1866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 14:49:32.289199 1881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:32.290544 1866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:49:32.290731 1866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "4247e4b83d6349e28d87ea29b15604b2"
format_stamp: "Formatted at 2026-06-19 14:49:32 on dist-test-slave-p3ml"
server_key: "55d5ad46f572c6e6854b04e49f25598f"
server_key_iv: "f43142b97a25fe7397e7799880c89194"
server_key_version: "encryptionkey@0"
I20260619 14:49:32.290923 1866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:32.304113 1866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:32.307852 1866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:32.308133 1866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:32.309036 1866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:32.310281 1866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:32.310382 1866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:32.310559 1866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:32.310637 1866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:32.343080 1866 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:33089
I20260619 14:49:32.343153 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:33089 every 8 connection(s)
I20260619 14:49:32.344307 1866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:49:32.346191 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 1866
I20260619 14:49:32.346388 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:49:32.346935 964 external_mini_cluster.cc:1428] Setting key 7fff876cdf58ecccaf612eceb50f73a5
Jun 19 14:49:32 dist-test-slave-p3ml krb5kdc[1493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:32.365000 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.346895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:56103 (local address 127.0.241.62:37097)
0619 14:49:32.347074 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.347081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.348646 (+ 1565us) server_negotiation.cc:407] Connection header received
0619 14:49:32.350846 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.350851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.350934 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.351050 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:32.352538 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.353243 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.354760 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.355163 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.355294 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.358665 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:32.358670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:32.358673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:32.358703 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:32.361523 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.362740 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.362749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.362752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.362867 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.363309 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.363313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.363315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.363641 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:32.363796 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.364433 (+ 637us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.364649 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:49:32.365975 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.347563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37097 (local address 127.0.241.3:56103)
0619 14:49:32.348474 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.348528 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:32.350606 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:32.351279 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:32.351287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:32.351845 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:32.352379 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:32.352395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.353398 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.353404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:32.354603 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:32.354613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.355374 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.355383 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:32.355642 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.356301 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:32.356321 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:32.358474 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:32.361677 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:32.361685 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:32.361701 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:32.362613 (+ 912us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:32.362985 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:32.362990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:32.362994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:32.363165 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:32.363771 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:32.363778 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:32.364096 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 14:49:32.364350 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 14:49:32.364854 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":360,"threads_started":1}
I20260619 14:49:32.367314 1994 heartbeater.cc:347] Connected to a master server at 127.0.241.62:37097
I20260619 14:49:32.367648 1994 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:32.368348 1994 heartbeater.cc:511] Master 127.0.241.62:37097 requested a full tablet report, sending...
I20260619 14:49:32.369702 1538 ts_manager.cc:194] Registered new tserver with Master: 4247e4b83d6349e28d87ea29b15604b2 (127.0.241.3:33089)
I20260619 14:49:32.370491 1538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:56103
I20260619 14:49:32.374217 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:49:32.393199 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.380194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52584 (local address 127.0.241.62:37097)
0619 14:49:32.380345 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.380350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.380804 (+ 454us) server_negotiation.cc:407] Connection header received
0619 14:49:32.381009 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.381014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.381069 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.381169 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:32.382557 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.383250 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.384945 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.385366 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.385524 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.387828 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:32.387836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:32.387845 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:32.387877 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:32.390610 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.391267 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.391274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.391278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.391411 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.391808 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.391816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.391819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.392113 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:32.392364 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.392684 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.392886 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:49:32.399247 1538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:49:32.406131 1538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:32.430029 1727 heartbeater.cc:503] Master 127.0.241.62:37097 was elected leader, sending a full tablet report...
I20260619 14:49:32.432551 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.417782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45243 (local address 127.0.0.1:40482)
0619 14:49:32.424236 (+ 6454us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.424249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:32.424397 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:32.425339 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:32.425346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:32.425383 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:32.425871 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:32.425885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.427728 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.427735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:32.429390 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:32.429412 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.431568 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.431578 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:32.431786 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.431847 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 14:49:32.431963 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 14:49:32.432165 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6366,"mutex_wait_us":4,"thread_start_us":680,"threads_started":1}
I20260619 14:49:32.432855 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.418302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.0.241.1:45243)
0619 14:49:32.423935 (+ 5633us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.423941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.424358 (+ 417us) server_negotiation.cc:407] Connection header received
0619 14:49:32.424586 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.424596 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.424850 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.425048 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:32.426053 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.427617 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.429592 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.431115 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.431239 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.432296 (+ 1057us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.432365 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.432548 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5491,"thread_start_us":5368,"threads_started":1}
I20260619 14:49:32.434692 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.425239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:46673 (local address 127.0.0.1:33078)
0619 14:49:32.425875 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.425894 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:32.426019 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:32.426666 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:32.426670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:32.426693 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:32.427541 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:32.427557 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.429313 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.429320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:32.432082 (+ 2762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:32.432092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.434013 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.434018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:32.434160 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.434190 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:49:32.434285 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 14:49:32.434393 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":618,"thread_start_us":344,"threads_started":1}
I20260619 14:49:32.435331 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.424711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.0.241.2:46673)
0619 14:49:32.425208 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.425216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.425986 (+ 770us) server_negotiation.cc:407] Connection header received
0619 14:49:32.426152 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.426159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.426353 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.426547 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:32.427716 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.429167 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.432225 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.433603 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.433729 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.433911 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.434879 (+ 968us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.435046 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":256,"threads_started":1}
I20260619 14:49:32.436205 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.417774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:33089 (local address 127.0.0.1:49254)
0619 14:49:32.418493 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.418538 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:32.418701 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:32.426403 (+ 7702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:32.426407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:32.426429 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:32.427276 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:32.427288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.429266 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.429273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:32.431608 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:32.431622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.433616 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.433623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:32.433787 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.435192 (+ 1405us) client_negotiation.cc:769] Sending connection context
0619 14:49:32.435285 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:49:32.435838 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"spinlock_wait_cycles":12032,"thread_start_us":6271,"threads_started":1}
I20260619 14:49:32.436928 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.417882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.0.241.3:33089)
0619 14:49:32.425803 (+ 7921us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.425816 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.425851 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:49:32.425947 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.425958 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.426202 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.426404 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:32.427498 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.429102 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.431754 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.433415 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.433570 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.433787 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.436489 (+ 2702us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.436660 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6897,"thread_start_us":6768,"threads_started":1}
I20260619 14:49:32.436800 1662 tablet_service.cc:1467] Processing CreateTablet for tablet 96745bb7ed4d4081abb256873f157892 (DEFAULT_TABLE table=test-table [id=972358404f3d4ac887563ce5110c9e23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:32.438331 1662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96745bb7ed4d4081abb256873f157892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:32.439316 1796 tablet_service.cc:1467] Processing CreateTablet for tablet 96745bb7ed4d4081abb256873f157892 (DEFAULT_TABLE table=test-table [id=972358404f3d4ac887563ce5110c9e23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:32.439671 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 96745bb7ed4d4081abb256873f157892 (DEFAULT_TABLE table=test-table [id=972358404f3d4ac887563ce5110c9e23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:32.441005 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96745bb7ed4d4081abb256873f157892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:32.441033 1796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96745bb7ed4d4081abb256873f157892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:32.451551 2012 tablet_bootstrap.cc:492] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2: Bootstrap starting.
I20260619 14:49:32.454936 2012 tablet_bootstrap.cc:654] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:32.455785 2014 tablet_bootstrap.cc:492] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12: Bootstrap starting.
I20260619 14:49:32.457404 2012 log.cc:824] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:32.458876 2013 tablet_bootstrap.cc:492] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d: Bootstrap starting.
I20260619 14:49:32.459503 2014 tablet_bootstrap.cc:654] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:32.460713 2012 tablet_bootstrap.cc:492] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2: No bootstrap required, opened a new log
I20260619 14:49:32.460747 2014 log.cc:824] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:32.461094 2012 ts_tablet_manager.cc:1397] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 14:49:32.462428 2013 tablet_bootstrap.cc:654] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:32.463652 2013 log.cc:824] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:32.464293 2014 tablet_bootstrap.cc:492] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12: No bootstrap required, opened a new log
I20260619 14:49:32.464561 2014 ts_tablet_manager.cc:1397] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 14:49:32.466446 2013 tablet_bootstrap.cc:492] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d: No bootstrap required, opened a new log
I20260619 14:49:32.466753 2013 ts_tablet_manager.cc:1397] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260619 14:49:32.469882 2012 raft_consensus.cc:348] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4247e4b83d6349e28d87ea29b15604b2" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33089 } } peers { permanent_uuid: "b8a20593f419418da9e3ee1058149f5d" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45243 } } peers { permanent_uuid: "564e7c1a1ee34b2a859f064a3827bf12" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 46673 } }
I20260619 14:49:32.470238 2012 raft_consensus.cc:374] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:32.470331 2012 raft_consensus.cc:729] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4247e4b83d6349e28d87ea29b15604b2, State: Initialized, Role: FOLLOWER
I20260619 14:49:32.470932 2012 consensus_queue.cc:260] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4247e4b83d6349e28d87ea29b15604b2" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33089 } } peers { permanent_uuid: "b8a20593f419418da9e3ee1058149f5d" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45243 } } peers { permanent_uuid: "564e7c1a1ee34b2a859f064a3827bf12" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 46673 } }
I20260619 14:49:32.472035 1994 heartbeater.cc:503] Master 127.0.241.62:37097 was elected leader, sending a full tablet report...
I20260619 14:49:32.472735 2012 ts_tablet_manager.cc:1428] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20260619 14:49:32.475106 2014 raft_consensus.cc:348] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4247e4b83d6349e28d87ea29b15604b2" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33089 } } peers { permanent_uuid: "b8a20593f419418da9e3ee1058149f5d" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45243 } } peers { permanent_uuid: "564e7c1a1ee34b2a859f064a3827bf12" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 46673 } }
I20260619 14:49:32.475612 2014 raft_consensus.cc:374] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:32.475711 2014 raft_consensus.cc:729] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 564e7c1a1ee34b2a859f064a3827bf12, State: Initialized, Role: FOLLOWER
I20260619 14:49:32.476420 2014 consensus_queue.cc:260] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4247e4b83d6349e28d87ea29b15604b2" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33089 } } peers { permanent_uuid: "b8a20593f419418da9e3ee1058149f5d" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45243 } } peers { permanent_uuid: "564e7c1a1ee34b2a859f064a3827bf12" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 46673 } }
I20260619 14:49:32.477439 2013 raft_consensus.cc:348] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4247e4b83d6349e28d87ea29b15604b2" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33089 } } peers { permanent_uuid: "b8a20593f419418da9e3ee1058149f5d" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45243 } } peers { permanent_uuid: "564e7c1a1ee34b2a859f064a3827bf12" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 46673 } }
I20260619 14:49:32.477957 2013 raft_consensus.cc:374] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:32.478080 2013 raft_consensus.cc:729] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8a20593f419418da9e3ee1058149f5d, State: Initialized, Role: FOLLOWER
I20260619 14:49:32.478775 2013 consensus_queue.cc:260] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4247e4b83d6349e28d87ea29b15604b2" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33089 } } peers { permanent_uuid: "b8a20593f419418da9e3ee1058149f5d" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45243 } } peers { permanent_uuid: "564e7c1a1ee34b2a859f064a3827bf12" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 46673 } }
I20260619 14:49:32.480899 2013 ts_tablet_manager.cc:1428] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260619 14:49:32.481884 1861 heartbeater.cc:503] Master 127.0.241.62:37097 was elected leader, sending a full tablet report...
I20260619 14:49:32.482430 2014 ts_tablet_manager.cc:1428] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12: Time spent starting tablet: real 0.018s user 0.011s sys 0.007s
I20260619 14:49:32.562273 2020 raft_consensus.cc:482] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:32.562592 2020 raft_consensus.cc:504] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4247e4b83d6349e28d87ea29b15604b2" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33089 } } peers { permanent_uuid: "b8a20593f419418da9e3ee1058149f5d" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45243 } } peers { permanent_uuid: "564e7c1a1ee34b2a859f064a3827bf12" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 46673 } }
I20260619 14:49:32.564342 2020 leader_election.cc:288] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4247e4b83d6349e28d87ea29b15604b2 (127.0.241.3:33089), 564e7c1a1ee34b2a859f064a3827bf12 (127.0.241.2:46673)
I20260619 14:49:32.571157 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.564742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:33399 (local address 127.0.241.3:33089)
0619 14:49:32.564986 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.564992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.565280 (+ 288us) server_negotiation.cc:407] Connection header received
0619 14:49:32.565508 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.565517 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.565618 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.565714 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:32.566303 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.567109 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.569344 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.570221 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.570360 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.570468 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.570744 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.570865 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:49:32.571206 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.564564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:33089 (local address 127.0.241.1:33399)
0619 14:49:32.565134 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.565150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:32.565331 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:32.565790 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:32.565795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:32.565828 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:32.566167 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:32.566175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.567234 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.567241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:32.569192 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:32.569204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.570427 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.570432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:32.570588 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.570611 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:49:32.570720 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:49:32.570845 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":303,"threads_started":1}
I20260619 14:49:32.572024 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.565507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:60935 (local address 127.0.241.2:46673)
0619 14:49:32.565715 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.565722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.565989 (+ 267us) server_negotiation.cc:407] Connection header received
0619 14:49:32.566182 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.566187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.566257 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.566354 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:32.566913 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.567698 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.569815 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.570878 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.571000 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.571105 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.571659 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.571781 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:49:32.572074 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.565254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:46673 (local address 127.0.241.1:60935)
0619 14:49:32.565879 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.565895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:32.566042 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:32.566368 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:32.566372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:32.566393 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:32.566765 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:32.566775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.567830 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.567839 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:32.569603 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:32.569615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.571280 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.571288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:32.571509 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.571539 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:49:32.571626 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:49:32.571742 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":325,"threads_started":1}
I20260619 14:49:32.572041 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96745bb7ed4d4081abb256873f157892" candidate_uuid: "b8a20593f419418da9e3ee1058149f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4247e4b83d6349e28d87ea29b15604b2" is_pre_election: true
I20260619 14:49:32.572603 1949 raft_consensus.cc:2454] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8a20593f419418da9e3ee1058149f5d in term 0.
I20260619 14:49:32.573100 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96745bb7ed4d4081abb256873f157892" candidate_uuid: "b8a20593f419418da9e3ee1058149f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564e7c1a1ee34b2a859f064a3827bf12" is_pre_election: true
I20260619 14:49:32.573289 1617 leader_election.cc:302] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4247e4b83d6349e28d87ea29b15604b2, b8a20593f419418da9e3ee1058149f5d; no voters:
I20260619 14:49:32.573599 1816 raft_consensus.cc:2454] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8a20593f419418da9e3ee1058149f5d in term 0.
I20260619 14:49:32.573688 2020 raft_consensus.cc:2781] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:32.573822 2020 raft_consensus.cc:482] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:32.573910 2020 raft_consensus.cc:3037] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:32.576246 2020 raft_consensus.cc:504] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4247e4b83d6349e28d87ea29b15604b2" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33089 } } peers { permanent_uuid: "b8a20593f419418da9e3ee1058149f5d" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45243 } } peers { permanent_uuid: "564e7c1a1ee34b2a859f064a3827bf12" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 46673 } }
I20260619 14:49:32.576824 2020 leader_election.cc:288] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [CANDIDATE]: Term 1 election: Requested vote from peers 4247e4b83d6349e28d87ea29b15604b2 (127.0.241.3:33089), 564e7c1a1ee34b2a859f064a3827bf12 (127.0.241.2:46673)
I20260619 14:49:32.577239 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96745bb7ed4d4081abb256873f157892" candidate_uuid: "b8a20593f419418da9e3ee1058149f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4247e4b83d6349e28d87ea29b15604b2"
I20260619 14:49:32.577303 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96745bb7ed4d4081abb256873f157892" candidate_uuid: "b8a20593f419418da9e3ee1058149f5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "564e7c1a1ee34b2a859f064a3827bf12"
I20260619 14:49:32.577445 1949 raft_consensus.cc:3037] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:32.577490 1816 raft_consensus.cc:3037] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:32.580051 1816 raft_consensus.cc:2454] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8a20593f419418da9e3ee1058149f5d in term 1.
I20260619 14:49:32.580185 1949 raft_consensus.cc:2454] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8a20593f419418da9e3ee1058149f5d in term 1.
I20260619 14:49:32.580499 1617 leader_election.cc:302] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 564e7c1a1ee34b2a859f064a3827bf12, b8a20593f419418da9e3ee1058149f5d; no voters:
I20260619 14:49:32.580766 2020 raft_consensus.cc:2781] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:32.581197 2020 raft_consensus.cc:686] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [term 1 LEADER]: Becoming Leader. State: Replica: b8a20593f419418da9e3ee1058149f5d, State: Running, Role: LEADER
I20260619 14:49:32.581633 2020 consensus_queue.cc:237] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4247e4b83d6349e28d87ea29b15604b2" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33089 } } peers { permanent_uuid: "b8a20593f419418da9e3ee1058149f5d" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45243 } } peers { permanent_uuid: "564e7c1a1ee34b2a859f064a3827bf12" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 46673 } }
I20260619 14:49:32.586122 1537 catalog_manager.cc:5495] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d reported cstate change: term changed from 0 to 1, leader changed from <none> to b8a20593f419418da9e3ee1058149f5d (127.0.241.1). New cstate: current_term: 1 leader_uuid: "b8a20593f419418da9e3ee1058149f5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4247e4b83d6349e28d87ea29b15604b2" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8a20593f419418da9e3ee1058149f5d" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "564e7c1a1ee34b2a859f064a3827bf12" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 46673 } health_report { overall_health: UNKNOWN } } }
W20260619 14:49:32.595796 1995 tablet.cc:2369] T 96745bb7ed4d4081abb256873f157892 P 4247e4b83d6349e28d87ea29b15604b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:49:32 dist-test-slave-p3ml krb5kdc[1493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
W20260619 14:49:32.624364 1862 tablet.cc:2369] T 96745bb7ed4d4081abb256873f157892 P 564e7c1a1ee34b2a859f064a3827bf12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:49:32.625041 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.610167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.0.241.1:45243)
0619 14:49:32.610410 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.610420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.610845 (+ 425us) server_negotiation.cc:407] Connection header received
0619 14:49:32.610983 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.610987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.611074 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.611156 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:32.612609 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.613367 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.615039 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.615519 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.615614 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.619564 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:32.619577 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:32.619593 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:32.619624 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:32.622391 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.622959 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.622965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.622967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.623114 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.623574 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.623578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.623580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.623923 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:32.624169 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.624525 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.624741 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 14:49:32 dist-test-slave-p3ml krb5kdc[1493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880572, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260619 14:49:32.649483 1728 tablet.cc:2369] T 96745bb7ed4d4081abb256873f157892 P b8a20593f419418da9e3ee1058149f5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:49:32 dist-test-slave-p3ml krb5kdc[1493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880572, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:49:32.676369 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.656127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.0.241.1:45243)
0619 14:49:32.656353 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.656361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.656691 (+ 330us) server_negotiation.cc:407] Connection header received
0619 14:49:32.656903 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.656907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.656977 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.657085 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:32.658457 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.659191 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.660551 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.660957 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.661092 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.669220 (+ 8128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:32.669227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:32.669232 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:32.669278 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:32.672848 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.673900 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.673909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.673915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.674096 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.674492 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.674499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.674504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.674877 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:32.675192 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.675607 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.675890 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 14:49:32 dist-test-slave-p3ml krb5kdc[1493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:49:32 dist-test-slave-p3ml krb5kdc[1493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:49:32.726897 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.713725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.0.241.1:45243)
0619 14:49:32.713910 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.713919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.714269 (+ 350us) server_negotiation.cc:407] Connection header received
0619 14:49:32.714485 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.714491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.714576 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.714685 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:32.716096 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.716886 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.718219 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.718646 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.718783 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.722398 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:32.722403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:32.722406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:32.722437 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:32.724750 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.725332 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.725336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.725338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.725455 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.725750 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.725753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.725755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.725982 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:32.726157 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.726368 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.726524 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 14:49:32.728675 1642 tablet_service.cc:2908] Rejecting scan request for tablet 96745bb7ed4d4081abb256873f157892: Uninitialized: clean time has not yet been initialized
I20260619 14:49:32.732004 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 1599
I20260619 14:49:32.743643 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.732964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.0.241.1:45243)
0619 14:49:32.733179 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.733187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.733486 (+ 299us) server_negotiation.cc:407] Connection header received
0619 14:49:32.733599 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.733602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.733653 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.733726 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:32.734775 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.735427 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.736563 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.736954 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.737043 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.738930 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:32.738934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:32.738937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:32.738964 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:32.741518 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.742078 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.742082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.742084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.742194 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.742490 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.742493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.742495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.742708 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:32.742878 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.743102 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.743267 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:49:32.744638 1722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:32.839120 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 1599
I20260619 14:49:32.856086 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 1733
Jun 19 14:49:32 dist-test-slave-p3ml krb5kdc[1493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:49:32.870814 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.856772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.0.241.2:46673)
0619 14:49:32.856946 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.856952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.856989 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:49:32.857199 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.857209 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.857286 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.857390 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:32.858548 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.859371 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.860680 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.861069 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.861200 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.865311 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:32.865324 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:32.865339 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:32.865369 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:32.868143 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.868788 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.868796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.868798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.868916 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.869292 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.869299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.869302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.869655 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:32.869977 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.870205 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.870424 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:49:32.871518 1856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:32.957664 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 1733
I20260619 14:49:32.974557 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 1866
Jun 19 14:49:32 dist-test-slave-p3ml krb5kdc[1493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:49:32.988307 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.975301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.0.241.3:33089)
0619 14:49:32.975650 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:32.975655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:32.975672 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:49:32.975727 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:32.975731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:32.975790 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:32.975875 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:32.977039 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.977724 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.979099 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:32.979543 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:32.979645 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:32.983551 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:32.983570 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:32.983593 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:32.983641 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:32.986009 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.986571 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.986577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.986579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.986704 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:32.987004 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:32.987006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:32.987008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:32.987317 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:32.987587 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:32.987820 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 14:49:32.988002 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:49:32.988906 1989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:33.073638 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 1866
I20260619 14:49:33.090269 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 1506
I20260619 14:49:33.091368 1568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:33.195660 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 1506
2026-06-19T14:49:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4048 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:49:33 dist-test-slave-p3ml krb5kdc[2042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:49:33 dist-test-slave-p3ml krb5kdc[2042](info): set up 2 sockets
Jun 19 14:49:33 dist-test-slave-p3ml krb5kdc[2042](info): commencing operation
krb5kdc: starting...
W20260619 14:49:34.275494 964 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.004s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:49:34 dist-test-slave-p3ml krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:49:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:49:34.448305 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33137
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:33137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:34.751660 2055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:34.752027 2055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:34.752214 2055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:34.759912 2055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:34.760016 2055 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:49:34.760056 2055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:34.760108 2055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:34.760157 2055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:34.760211 2055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:34.770361 2055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:33137
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:34.771615 2055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:34.772984 2055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:34.784626 2061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:34.784682 2060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:34.784705 2063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:34.788859 2055 server_base.cc:1034] running on GCE node
I20260619 14:49:34.789601 2055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:34.790930 2055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:34.792151 2055 hybrid_clock.cc:648] HybridClock initialized: now 1781880574792093 us; error 30 us; skew 500 ppm
I20260619 14:49:34.792587 2055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:34 dist-test-slave-p3ml krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:34.795586 2055 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:49:34.796983 2055 webserver.cc:460] Webserver started at http://127.0.241.62:42707/ using document root <none> and password file <none>
I20260619 14:49:34.797636 2055 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:34.797715 2055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:34.798002 2055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:34.800801 2055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "88feb61578ad4befae45a8ee963b10e3"
format_stamp: "Formatted at 2026-06-19 14:49:34 on dist-test-slave-p3ml"
server_key: "3a5793df1be09530d4648fdb983a9241"
server_key_iv: "99e0bc5d9edc0677d5865110e4761ba3"
server_key_version: "encryptionkey@0"
I20260619 14:49:34.801518 2055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "88feb61578ad4befae45a8ee963b10e3"
format_stamp: "Formatted at 2026-06-19 14:49:34 on dist-test-slave-p3ml"
server_key: "3a5793df1be09530d4648fdb983a9241"
server_key_iv: "99e0bc5d9edc0677d5865110e4761ba3"
server_key_version: "encryptionkey@0"
I20260619 14:49:34.806445 2055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 14:49:34.809640 2070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:34.810869 2055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:49:34.811069 2055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "88feb61578ad4befae45a8ee963b10e3"
format_stamp: "Formatted at 2026-06-19 14:49:34 on dist-test-slave-p3ml"
server_key: "3a5793df1be09530d4648fdb983a9241"
server_key_iv: "99e0bc5d9edc0677d5865110e4761ba3"
server_key_version: "encryptionkey@0"
I20260619 14:49:34.811275 2055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:34.826113 2055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:34.829788 2055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:34.830057 2055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:34.851094 2055 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:33137
I20260619 14:49:34.851099 2131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:33137 every 8 connection(s)
I20260619 14:49:34.852478 2055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:34.853955 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 2055
I20260619 14:49:34.854182 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:49:34.855003 964 external_mini_cluster.cc:1428] Setting key 107db9f531cabf1afe4ea5f1b210b86b
I20260619 14:49:34.857283 2132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 14:49:34 dist-test-slave-p3ml krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:34.870338 2132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3: Bootstrap starting.
I20260619 14:49:34.874496 2132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:34.875836 2132 log.cc:824] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:34.878099 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:34.857036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56980 (local address 127.0.241.62:33137)
0619 14:49:34.857787 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:34.857798 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:34.857836 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:49:34.859407 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:34.859478 (+ 71us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:34.859865 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:34.860189 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:34.861878 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:34.863193 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:34.864898 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:34.865524 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:34.865654 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:34.870289 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:34.870299 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:34.870322 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:34.870369 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:34.874113 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:34.874789 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:34.874799 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:34.874809 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:34.875022 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:34.875355 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:34.875359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:34.875362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:34.876110 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:34.876419 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:34.876739 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 14:49:34.877102 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":367,"threads_started":1}
I20260619 14:49:34.878788 2132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3: No bootstrap required, opened a new log
I20260619 14:49:34.885829 2132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88feb61578ad4befae45a8ee963b10e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33137 } }
I20260619 14:49:34.886188 2132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:34.886291 2132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88feb61578ad4befae45a8ee963b10e3, State: Initialized, Role: FOLLOWER
I20260619 14:49:34.886818 2132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88feb61578ad4befae45a8ee963b10e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33137 } }
I20260619 14:49:34.887058 2132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:34.887197 2132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:34.887365 2132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:34.889441 2132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88feb61578ad4befae45a8ee963b10e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33137 } }
I20260619 14:49:34.889947 2132 leader_election.cc:302] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88feb61578ad4befae45a8ee963b10e3; no voters:
I20260619 14:49:34.890522 2132 leader_election.cc:288] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:49:34.890648 2137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:34.891342 2137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [term 1 LEADER]: Becoming Leader. State: Replica: 88feb61578ad4befae45a8ee963b10e3, State: Running, Role: LEADER
I20260619 14:49:34.891922 2137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88feb61578ad4befae45a8ee963b10e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33137 } }
I20260619 14:49:34.892681 2132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:34.895058 2139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88feb61578ad4befae45a8ee963b10e3. Latest consensus state: current_term: 1 leader_uuid: "88feb61578ad4befae45a8ee963b10e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88feb61578ad4befae45a8ee963b10e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33137 } } }
I20260619 14:49:34.894976 2138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88feb61578ad4befae45a8ee963b10e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88feb61578ad4befae45a8ee963b10e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33137 } } }
I20260619 14:49:34.895460 2139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:34.895538 2138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:34.897157 2144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:34.902536 2144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:34.913345 2144 catalog_manager.cc:1347] Generated new cluster ID: fbf8b9350bd64716bd5960260b4d4e41
I20260619 14:49:34.913460 2144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:34.930214 2144 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:34.932034 2144 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:34.939334 2144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3: Generated new TSK 0
I20260619 14:49:34.940117 2144 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:49:34.995913 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33137
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:35.325098 2158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:35.325361 2158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:35.325476 2158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:35.332752 2158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:35.332919 2158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:35.333074 2158 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:49:35.333230 2158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:49:35.342767 2158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.241.62:33137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:35.344074 2158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:35.345382 2158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:35.358189 2163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:35.358225 2166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:35.358134 2164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:35.359246 2158 server_base.cc:1034] running on GCE node
I20260619 14:49:35.359900 2158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:35.360778 2158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:35.361985 2158 hybrid_clock.cc:648] HybridClock initialized: now 1781880575361907 us; error 56 us; skew 500 ppm
I20260619 14:49:35.362263 2158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:35 dist-test-slave-p3ml krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:35.365422 2158 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:49:35.366791 2158 webserver.cc:460] Webserver started at http://127.0.241.1:44975/ using document root <none> and password file <none>
I20260619 14:49:35.367420 2158 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:35.367585 2158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:35.367849 2158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:35.370618 2158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "5a4b9ff6c8224411a3a8806317a5c909"
format_stamp: "Formatted at 2026-06-19 14:49:35 on dist-test-slave-p3ml"
server_key: "f2f8af825f046920930d59b0038700b7"
server_key_iv: "ae9267873d83c7d739c45c6ce8e25fe7"
server_key_version: "encryptionkey@0"
I20260619 14:49:35.371337 2158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "5a4b9ff6c8224411a3a8806317a5c909"
format_stamp: "Formatted at 2026-06-19 14:49:35 on dist-test-slave-p3ml"
server_key: "f2f8af825f046920930d59b0038700b7"
server_key_iv: "ae9267873d83c7d739c45c6ce8e25fe7"
server_key_version: "encryptionkey@0"
I20260619 14:49:35.376297 2158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 14:49:35.379664 2173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:35.380973 2158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:49:35.381141 2158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "5a4b9ff6c8224411a3a8806317a5c909"
format_stamp: "Formatted at 2026-06-19 14:49:35 on dist-test-slave-p3ml"
server_key: "f2f8af825f046920930d59b0038700b7"
server_key_iv: "ae9267873d83c7d739c45c6ce8e25fe7"
server_key_version: "encryptionkey@0"
I20260619 14:49:35.381349 2158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:35.393304 2158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:35.397791 2158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:35.398051 2158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:35.400763 2158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:35.400869 2158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:35.401073 2158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:35.401149 2158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:49:35 dist-test-slave-p3ml krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:35.423748 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:35.401212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:52489 (local address 127.0.241.62:33137)
0619 14:49:35.401640 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:35.401646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:35.402860 (+ 1214us) server_negotiation.cc:407] Connection header received
0619 14:49:35.405041 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:35.405049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:35.405140 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:35.405272 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:35.406913 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.408106 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.410899 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.411567 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.411695 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:35.416868 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:35.416875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:35.416880 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:35.416921 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:35.419678 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:35.420622 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:35.420630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:35.420637 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:35.420805 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:35.421307 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:35.421313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:35.421317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:35.421707 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:35.421858 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:35.422876 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0619 14:49:35.423171 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":220,"threads_started":1}
I20260619 14:49:35.424451 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:35.401838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33137 (local address 127.0.241.1:52489)
0619 14:49:35.402690 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:35.402734 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:35.404773 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:35.405460 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:35.405466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:35.405954 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:35.406734 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:35.406751 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.408298 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.408309 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:35.410706 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:35.410722 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.411811 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.411819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:35.412198 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:35.413114 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:35.413162 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:35.416614 (+ 3452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:35.419843 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:35.419856 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:35.419877 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:35.420475 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:35.420947 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:35.420953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:35.420961 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:35.421187 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:35.421852 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:35.421859 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:35.422376 (+ 517us) client_negotiation.cc:769] Sending connection context
0619 14:49:35.422770 (+ 394us) client_negotiation.cc:240] Negotiation successful
0619 14:49:35.423165 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":378,"threads_started":1}
I20260619 14:49:35.447937 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:35.433519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:50855 (local address 127.0.241.62:33137)
0619 14:49:35.433685 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:35.433689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:35.434260 (+ 571us) server_negotiation.cc:407] Connection header received
0619 14:49:35.434458 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:35.434463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:35.434530 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:35.434628 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:35.435577 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.436412 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.438556 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.439326 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.440389 (+ 1063us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:35.442532 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:35.442541 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:35.442547 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:35.442587 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:35.445381 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:35.446033 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:35.446040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:35.446044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:35.446160 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:35.446506 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:35.446513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:35.446516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:35.446761 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:35.446905 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:35.447342 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 14:49:35.447577 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:49:35.448369 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:35.433554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33137 (local address 127.0.241.1:50855)
0619 14:49:35.434159 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:35.434177 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:35.434328 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:35.434651 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:35.434655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:35.435117 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:35.435384 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:35.435392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.436610 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.436613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:35.438294 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:35.438331 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.439646 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.439655 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:35.439937 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:35.440788 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:35.440815 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:35.442292 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:35.445538 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:35.445546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:35.445549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:35.445922 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:35.446265 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:35.446270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:35.446273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:35.446398 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:35.446872 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:35.446876 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:35.447076 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 14:49:35.447223 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 14:49:35.447577 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":361,"threads_started":1}
I20260619 14:49:35.455547 2158 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:33495
I20260619 14:49:35.455654 2293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:33495 every 8 connection(s)
I20260619 14:49:35.456737 2158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:35.458767 2294 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33137
I20260619 14:49:35.458945 2294 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:35.459600 2294 heartbeater.cc:511] Master 127.0.241.62:33137 requested a full tablet report, sending...
I20260619 14:49:35.461393 2087 ts_manager.cc:194] Registered new tserver with Master: 5a4b9ff6c8224411a3a8806317a5c909 (127.0.241.1:33495)
I20260619 14:49:35.462819 2087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:52489
I20260619 14:49:35.465298 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 2158
I20260619 14:49:35.465508 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:49:35.466112 964 external_mini_cluster.cc:1428] Setting key d8d285a8752e430ab927739a29ad2a9d
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:49:35.504727 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33137
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:35.827980 2298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:35.828241 2298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:35.828332 2298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:35.835209 2298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:35.835300 2298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:35.835379 2298 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:49:35.835530 2298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:49:35.844924 2298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.241.62:33137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:35.846129 2298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:35.847404 2298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:35.861236 2304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:35.861275 2303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:35.864424 2306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:35.865061 2298 server_base.cc:1034] running on GCE node
I20260619 14:49:35.865732 2298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:35.866783 2298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:35.868005 2298 hybrid_clock.cc:648] HybridClock initialized: now 1781880575867924 us; error 60 us; skew 500 ppm
I20260619 14:49:35.868324 2298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:35 dist-test-slave-p3ml krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:35.871810 2298 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:49:35.873231 2298 webserver.cc:460] Webserver started at http://127.0.241.2:35447/ using document root <none> and password file <none>
I20260619 14:49:35.873878 2298 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:35.873960 2298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:35.874207 2298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:35.876965 2298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "763ccb8f1fa7443cba7124d1697fb909"
format_stamp: "Formatted at 2026-06-19 14:49:35 on dist-test-slave-p3ml"
server_key: "2a70fb98f0096a0f65faebd0546e8742"
server_key_iv: "f031965585b90be2ccdf5e58feed3a1b"
server_key_version: "encryptionkey@0"
I20260619 14:49:35.877723 2298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "763ccb8f1fa7443cba7124d1697fb909"
format_stamp: "Formatted at 2026-06-19 14:49:35 on dist-test-slave-p3ml"
server_key: "2a70fb98f0096a0f65faebd0546e8742"
server_key_iv: "f031965585b90be2ccdf5e58feed3a1b"
server_key_version: "encryptionkey@0"
I20260619 14:49:35.882704 2298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 14:49:35.886135 2313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:35.887358 2298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:49:35.887568 2298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "763ccb8f1fa7443cba7124d1697fb909"
format_stamp: "Formatted at 2026-06-19 14:49:35 on dist-test-slave-p3ml"
server_key: "2a70fb98f0096a0f65faebd0546e8742"
server_key_iv: "f031965585b90be2ccdf5e58feed3a1b"
server_key_version: "encryptionkey@0"
I20260619 14:49:35.887787 2298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:35.900991 2298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:35.904711 2298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:35.904976 2298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:35.907418 2298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:35.907590 2298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:35.907747 2298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:35.907832 2298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:49:35 dist-test-slave-p3ml krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:35.932220 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:35.908584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33137 (local address 127.0.241.2:41661)
0619 14:49:35.909424 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:35.909469 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:35.911556 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:35.912297 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:35.912309 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:35.913062 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:35.913843 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:35.913864 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.915572 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.915580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:35.917899 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:35.917910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.918806 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.918813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:35.919127 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:35.920055 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:35.920092 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:35.923001 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:35.926873 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:35.926884 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:35.926904 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:35.927628 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:35.928095 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:35.928100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:35.928104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:35.928473 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:35.929249 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:35.929260 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:35.929754 (+ 494us) client_negotiation.cc:769] Sending connection context
0619 14:49:35.930060 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 14:49:35.930582 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":381,"threads_started":1}
I20260619 14:49:35.933588 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:35.908075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:41661 (local address 127.0.241.62:33137)
0619 14:49:35.908266 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:35.908272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:35.909608 (+ 1336us) server_negotiation.cc:407] Connection header received
0619 14:49:35.911821 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:35.911827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:35.911923 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:35.912036 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:35.914052 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.915329 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.918094 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.918559 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.918691 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:35.923195 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:35.923200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:35.923205 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:35.923242 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:35.926728 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:35.927807 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:35.927814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:35.927820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:35.927970 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:35.928633 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:35.928636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:35.928639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:35.929019 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:35.929183 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:35.932999 (+ 3816us) server_negotiation.cc:299] Negotiation successful
0619 14:49:35.933243 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 14:49:35.951123 2298 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:40943
I20260619 14:49:35.951174 2432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:40943 every 8 connection(s)
I20260619 14:49:35.952807 2298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:49:35.954972 2433 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33137
I20260619 14:49:35.955191 2433 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:35.955884 2433 heartbeater.cc:511] Master 127.0.241.62:33137 requested a full tablet report, sending...
I20260619 14:49:35.957093 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:35.940213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:59963 (local address 127.0.241.62:33137)
0619 14:49:35.940412 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:35.940418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:35.944160 (+ 3742us) server_negotiation.cc:407] Connection header received
0619 14:49:35.944411 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:35.944426 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:35.944488 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:35.944591 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:35.945483 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.946158 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.947412 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.947935 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.948059 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:35.950577 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:35.950586 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:35.950594 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:35.950642 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:35.954269 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:35.955061 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:35.955066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:35.955069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:35.955212 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:35.955647 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:35.955654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:35.955657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:35.955994 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:35.956125 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:35.956525 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 14:49:35.956733 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:49:35.957162 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:35.940238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33137 (local address 127.0.241.2:59963)
0619 14:49:35.944020 (+ 3782us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:35.944046 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:35.944221 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:35.944617 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:35.944621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:35.945070 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:35.945343 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:35.945351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.946287 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.946294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:35.947279 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:35.947289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:35.948268 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:35.948275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:35.948544 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:35.949484 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:35.949499 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:35.950367 (+ 868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:35.954417 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:35.954423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:35.954427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:35.954929 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:35.955326 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:35.955331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:35.955335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:35.955542 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:35.956095 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:35.956099 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:35.956270 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 14:49:35.956436 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 14:49:35.956751 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3599,"thread_start_us":3420,"threads_started":1}
I20260619 14:49:35.957269 2087 ts_manager.cc:194] Registered new tserver with Master: 763ccb8f1fa7443cba7124d1697fb909 (127.0.241.2:40943)
I20260619 14:49:35.958355 2087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:41661
I20260619 14:49:35.961459 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 2298
I20260619 14:49:35.961637 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:49:35.962096 964 external_mini_cluster.cc:1428] Setting key 005ad1b2da2340254fd0c1fa7e44ad68
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:49:36.000994 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33137
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:35637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:36.315832 2437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:36.316147 2437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:36.316275 2437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:36.323099 2437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:36.323194 2437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:36.323274 2437 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:49:36.323344 2437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:49:36.332813 2437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:35637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.241.62:33137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:36.333968 2437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:36.335309 2437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:36.348337 2443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:36.348451 2442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:36.348697 2445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:36.349340 2437 server_base.cc:1034] running on GCE node
I20260619 14:49:36.349954 2437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:36.350804 2437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:36.352028 2437 hybrid_clock.cc:648] HybridClock initialized: now 1781880576351961 us; error 62 us; skew 500 ppm
I20260619 14:49:36.352306 2437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:36 dist-test-slave-p3ml krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:36.355253 2437 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:49:36.356624 2437 webserver.cc:460] Webserver started at http://127.0.241.3:46791/ using document root <none> and password file <none>
I20260619 14:49:36.357301 2437 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:36.357436 2437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:36.357748 2437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:36.360564 2437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "3c9da3df10584819b4241855f5857bc1"
format_stamp: "Formatted at 2026-06-19 14:49:36 on dist-test-slave-p3ml"
server_key: "cb34eb934412010242cf39aeeac98468"
server_key_iv: "767e8c70dec402923ab570a322a0e357"
server_key_version: "encryptionkey@0"
I20260619 14:49:36.361330 2437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "3c9da3df10584819b4241855f5857bc1"
format_stamp: "Formatted at 2026-06-19 14:49:36 on dist-test-slave-p3ml"
server_key: "cb34eb934412010242cf39aeeac98468"
server_key_iv: "767e8c70dec402923ab570a322a0e357"
server_key_version: "encryptionkey@0"
I20260619 14:49:36.366226 2437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 14:49:36.369527 2452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:36.370704 2437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:49:36.370864 2437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "3c9da3df10584819b4241855f5857bc1"
format_stamp: "Formatted at 2026-06-19 14:49:36 on dist-test-slave-p3ml"
server_key: "cb34eb934412010242cf39aeeac98468"
server_key_iv: "767e8c70dec402923ab570a322a0e357"
server_key_version: "encryptionkey@0"
I20260619 14:49:36.371090 2437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:36.387063 2437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:36.391585 2437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:36.391846 2437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:36.394544 2437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:36.394650 2437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:36.394815 2437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:36.394891 2437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:49:36 dist-test-slave-p3ml krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:36.422147 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.395191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:52601 (local address 127.0.241.62:33137)
0619 14:49:36.395392 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:36.395398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:36.396990 (+ 1592us) server_negotiation.cc:407] Connection header received
0619 14:49:36.400142 (+ 3152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:36.400151 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:36.400250 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:36.400367 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:36.403128 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.404192 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.407372 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.407903 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.408050 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.413049 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:36.413056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:36.413060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:36.413102 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:36.417041 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:36.417838 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:36.417847 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:36.417852 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:36.418038 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:36.418711 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:36.418715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:36.418719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:36.419144 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:36.419292 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:36.421584 (+ 2292us) server_negotiation.cc:299] Negotiation successful
0619 14:49:36.421811 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:49:36.422585 2461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.395879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33137 (local address 127.0.241.3:52601)
0619 14:49:36.396804 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:36.396854 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:36.399772 (+ 2918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:36.400683 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:36.400696 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:36.401689 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:36.402843 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.402873 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.404357 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.404364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.407206 (+ 2842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.407228 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.408354 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.408359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.408632 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.409511 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:36.409547 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:36.412818 (+ 3271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:36.417163 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:36.417169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:36.417180 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:36.417704 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:36.418196 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:36.418201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:36.418206 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:36.418533 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:36.419398 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:36.419409 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:36.419990 (+ 581us) client_negotiation.cc:769] Sending connection context
0619 14:49:36.420365 (+ 375us) client_negotiation.cc:240] Negotiation successful
0619 14:49:36.421079 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":352,"threads_started":1}
I20260619 14:49:36.440562 2437 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:43181
I20260619 14:49:36.440649 2571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:43181 every 8 connection(s)
I20260619 14:49:36.441787 2437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:49:36.443046 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.429676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:47331 (local address 127.0.241.62:33137)
0619 14:49:36.429900 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:36.429916 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:36.430476 (+ 560us) server_negotiation.cc:407] Connection header received
0619 14:49:36.430710 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:36.430715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:36.430792 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:36.430898 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:36.432023 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.433069 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.434372 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.435079 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.435197 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.437268 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:36.437276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:36.437284 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:36.437326 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:36.440252 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:36.441107 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:36.441113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:36.441117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:36.441240 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:36.441598 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:36.441605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:36.441609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:36.441854 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:36.442007 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:36.442463 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 14:49:36.442674 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:49:36.443177 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.429687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33137 (local address 127.0.241.3:47331)
0619 14:49:36.430345 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:36.430371 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:36.430559 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:36.430925 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:36.430930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:36.431354 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:36.431867 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.431882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.433207 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.433213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.434224 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.434235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.435272 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.435280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.435486 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.436275 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:36.436291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:36.437088 (+ 797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:36.440383 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:36.440399 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:36.440406 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:36.440972 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:36.441349 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:36.441354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:36.441358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:36.441488 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:36.442056 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:36.442059 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:36.442224 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 14:49:36.442372 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 14:49:36.442697 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":348,"threads_started":1}
I20260619 14:49:36.444264 2572 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33137
I20260619 14:49:36.444502 2572 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:36.445194 2572 heartbeater.cc:511] Master 127.0.241.62:33137 requested a full tablet report, sending...
I20260619 14:49:36.446557 2086 ts_manager.cc:194] Registered new tserver with Master: 3c9da3df10584819b4241855f5857bc1 (127.0.241.3:43181)
I20260619 14:49:36.447499 2086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:52601
I20260619 14:49:36.449913 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 2437
I20260619 14:49:36.450054 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:49:36.450446 964 external_mini_cluster.cc:1428] Setting key e11ec1b96e382b2868e51384c0e3ae42
I20260619 14:49:36.453002 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:49:36.465399 2294 heartbeater.cc:503] Master 127.0.241.62:33137 was elected leader, sending a full tablet report...
Jun 19 14:49:36 dist-test-slave-p3ml krb5kdc[2042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880576, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:49:36 dist-test-slave-p3ml krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880576, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:36.500481 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.485890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.0.241.62:33137)
0619 14:49:36.486114 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:36.486121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:36.486402 (+ 281us) server_negotiation.cc:407] Connection header received
0619 14:49:36.486585 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:36.486590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:36.486650 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:36.486733 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:36.488441 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.489165 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.490557 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.490952 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.491055 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.495168 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:36.495173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:36.495176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:36.495207 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:36.497698 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:36.498586 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:36.498590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:36.498592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:36.498703 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:36.499067 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:36.499071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:36.499073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:36.499380 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:36.499652 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:36.499919 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 14:49:36.500091 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 14:49:36.504781 2086 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:49:36.511720 2086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:36.534490 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.523767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.0.241.2:40943)
0619 14:49:36.525010 (+ 1243us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:36.525020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:36.525050 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:49:36.525143 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:36.525150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:36.525445 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:36.525663 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:36.526945 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.528397 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.531830 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.533120 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.533253 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.533450 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:36.534008 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 14:49:36.534170 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":1192,"threads_started":1}
I20260619 14:49:36.534617 2587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.524948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33495 (local address 127.0.0.1:36810)
0619 14:49:36.525592 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:36.525610 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:36.525714 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:36.526872 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:36.526881 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:36.526912 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:36.527492 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.527508 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.529695 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.529701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.531979 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.531993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.533749 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.533756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.533944 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.533975 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:49:36.534068 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:49:36.534190 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":57,"thread_start_us":397,"threads_started":1}
I20260619 14:49:36.534348 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.523554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:40943 (local address 127.0.0.1:56060)
0619 14:49:36.524415 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:36.524461 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:36.524629 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:36.525960 (+ 1331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:36.525966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:36.525995 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:36.526458 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.526472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.528585 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.528594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.531657 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.531670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.533331 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.533337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.533522 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.533591 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 14:49:36.533719 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 14:49:36.533920 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":408,"thread_start_us":424,"threads_started":1}
I20260619 14:49:36.535207 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.525184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.0.241.1:33495)
0619 14:49:36.526216 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:36.526225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:36.526252 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:49:36.526361 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:36.526371 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:36.526620 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:36.527763 (+ 1143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:36.527954 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.529560 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.532152 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.533548 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.533693 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.533894 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:36.534435 (+ 541us) server_negotiation.cc:299] Negotiation successful
0619 14:49:36.534605 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":733,"threads_started":1}
I20260619 14:49:36.536064 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.525649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:43181 (local address 127.0.0.1:59722)
0619 14:49:36.526682 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:36.526698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:36.526795 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:36.528620 (+ 1825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:36.528626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:36.528657 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:36.529107 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.529117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.531319 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.531326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.533150 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.533161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.535199 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.535205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.535484 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.535514 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:49:36.535579 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:49:36.535713 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"spinlock_wait_cycles":7424,"thread_start_us":782,"threads_started":1}
I20260619 14:49:36.536855 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.526226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59722 (local address 127.0.241.3:43181)
0619 14:49:36.528101 (+ 1875us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:36.528109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:36.528138 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:49:36.528235 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:36.528242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:36.528437 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:36.529309 (+ 872us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:36.529587 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.531202 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.533367 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.534961 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.536108 (+ 1147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.536336 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:36.536405 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 14:49:36.536566 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1725,"thread_start_us":1552,"threads_started":1}
I20260619 14:49:36.539014 2363 tablet_service.cc:1467] Processing CreateTablet for tablet e34cf0b1d58c4a0e8e716809addf6270 (DEFAULT_TABLE table=test-table [id=3fb10f4986fe4017882f0bdbf95ab3e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:36.539073 2224 tablet_service.cc:1467] Processing CreateTablet for tablet e34cf0b1d58c4a0e8e716809addf6270 (DEFAULT_TABLE table=test-table [id=3fb10f4986fe4017882f0bdbf95ab3e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:36.540426 2224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34cf0b1d58c4a0e8e716809addf6270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:36.540447 2363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34cf0b1d58c4a0e8e716809addf6270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:36.540127 2502 tablet_service.cc:1467] Processing CreateTablet for tablet e34cf0b1d58c4a0e8e716809addf6270 (DEFAULT_TABLE table=test-table [id=3fb10f4986fe4017882f0bdbf95ab3e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:36.541378 2502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e34cf0b1d58c4a0e8e716809addf6270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:36.553187 2591 tablet_bootstrap.cc:492] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909: Bootstrap starting.
I20260619 14:49:36.555408 2592 tablet_bootstrap.cc:492] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909: Bootstrap starting.
I20260619 14:49:36.557067 2593 tablet_bootstrap.cc:492] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1: Bootstrap starting.
I20260619 14:49:36.558434 2591 tablet_bootstrap.cc:654] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:36.558925 2592 tablet_bootstrap.cc:654] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:36.559821 2591 log.cc:824] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:36.559940 2593 tablet_bootstrap.cc:654] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:36.560618 2592 log.cc:824] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:36.560961 2593 log.cc:824] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:36.563838 2593 tablet_bootstrap.cc:492] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1: No bootstrap required, opened a new log
I20260619 14:49:36.563949 2591 tablet_bootstrap.cc:492] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909: No bootstrap required, opened a new log
I20260619 14:49:36.564105 2592 tablet_bootstrap.cc:492] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909: No bootstrap required, opened a new log
I20260619 14:49:36.564116 2593 ts_tablet_manager.cc:1397] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:49:36.564210 2591 ts_tablet_manager.cc:1397] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260619 14:49:36.564344 2592 ts_tablet_manager.cc:1397] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 14:49:36.574743 2591 raft_consensus.cc:348] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.575222 2591 raft_consensus.cc:374] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:36.575327 2591 raft_consensus.cc:729] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763ccb8f1fa7443cba7124d1697fb909, State: Initialized, Role: FOLLOWER
I20260619 14:49:36.575169 2593 raft_consensus.cc:348] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.575169 2592 raft_consensus.cc:348] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.575598 2592 raft_consensus.cc:374] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:36.575598 2593 raft_consensus.cc:374] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:36.575708 2593 raft_consensus.cc:729] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9da3df10584819b4241855f5857bc1, State: Initialized, Role: FOLLOWER
I20260619 14:49:36.575830 2592 raft_consensus.cc:729] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9ff6c8224411a3a8806317a5c909, State: Initialized, Role: FOLLOWER
I20260619 14:49:36.576107 2591 consensus_queue.cc:260] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.576390 2592 consensus_queue.cc:260] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.576390 2593 consensus_queue.cc:260] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.577353 2433 heartbeater.cc:503] Master 127.0.241.62:33137 was elected leader, sending a full tablet report...
I20260619 14:49:36.577961 2591 ts_tablet_manager.cc:1428] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20260619 14:49:36.578871 2572 heartbeater.cc:503] Master 127.0.241.62:33137 was elected leader, sending a full tablet report...
I20260619 14:49:36.579541 2593 ts_tablet_manager.cc:1428] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260619 14:49:36.580971 2592 ts_tablet_manager.cc:1428] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260619 14:49:36.625968 2599 raft_consensus.cc:482] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:36.626302 2599 raft_consensus.cc:504] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.628165 2599 leader_election.cc:288] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 763ccb8f1fa7443cba7124d1697fb909 (127.0.241.2:40943), 3c9da3df10584819b4241855f5857bc1 (127.0.241.3:43181)
I20260619 14:49:36.635605 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.628566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:60823 (local address 127.0.241.2:40943)
0619 14:49:36.628763 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:36.628767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:36.629300 (+ 533us) server_negotiation.cc:407] Connection header received
0619 14:49:36.629502 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:36.629507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:36.629585 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:36.629688 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:36.630497 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.631529 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.633705 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.634679 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.634781 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.634854 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:36.635235 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 14:49:36.635323 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:49:36.635803 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.628496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:40943 (local address 127.0.241.1:60823)
0619 14:49:36.629174 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:36.629199 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:36.629343 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:36.629895 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:36.629900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:36.629948 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:36.630350 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.630362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.631647 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.631652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.633554 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.633565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.634873 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.634879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.635066 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.635100 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:49:36.635198 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:49:36.635325 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":348,"threads_started":1}
I20260619 14:49:36.636397 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.629564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:42607 (local address 127.0.241.3:43181)
0619 14:49:36.629815 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:36.629821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:36.630216 (+ 395us) server_negotiation.cc:407] Connection header received
0619 14:49:36.630359 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:36.630363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:36.630453 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:36.630542 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:36.631108 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.632023 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.634296 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.635379 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.635538 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.635637 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:36.636051 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 14:49:36.636194 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:49:36.636899 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.629297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:43181 (local address 127.0.241.1:42607)
0619 14:49:36.630118 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:36.630132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:36.630234 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:36.630568 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:36.630574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:36.630602 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:36.630957 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.630983 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.632154 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.632158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.634153 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.634166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.635631 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.635638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.635838 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.635884 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 14:49:36.636587 (+ 703us) client_negotiation.cc:240] Negotiation successful
0619 14:49:36.636693 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":479,"threads_started":1}
I20260619 14:49:36.636893 2383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34cf0b1d58c4a0e8e716809addf6270" candidate_uuid: "5a4b9ff6c8224411a3a8806317a5c909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763ccb8f1fa7443cba7124d1697fb909" is_pre_election: true
I20260619 14:49:36.637502 2383 raft_consensus.cc:2454] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9ff6c8224411a3a8806317a5c909 in term 0.
I20260619 14:49:36.637856 2522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34cf0b1d58c4a0e8e716809addf6270" candidate_uuid: "5a4b9ff6c8224411a3a8806317a5c909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9da3df10584819b4241855f5857bc1" is_pre_election: true
I20260619 14:49:36.638236 2174 leader_election.cc:302] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9ff6c8224411a3a8806317a5c909, 763ccb8f1fa7443cba7124d1697fb909; no voters:
I20260619 14:49:36.638340 2522 raft_consensus.cc:2454] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9ff6c8224411a3a8806317a5c909 in term 0.
I20260619 14:49:36.638589 2599 raft_consensus.cc:2781] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:36.638727 2599 raft_consensus.cc:482] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:36.638839 2599 raft_consensus.cc:3037] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:36.668114 2599 raft_consensus.cc:504] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.669019 2599 leader_election.cc:288] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [CANDIDATE]: Term 1 election: Requested vote from peers 763ccb8f1fa7443cba7124d1697fb909 (127.0.241.2:40943), 3c9da3df10584819b4241855f5857bc1 (127.0.241.3:43181)
I20260619 14:49:36.669625 2383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34cf0b1d58c4a0e8e716809addf6270" candidate_uuid: "5a4b9ff6c8224411a3a8806317a5c909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763ccb8f1fa7443cba7124d1697fb909"
I20260619 14:49:36.669674 2522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e34cf0b1d58c4a0e8e716809addf6270" candidate_uuid: "5a4b9ff6c8224411a3a8806317a5c909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9da3df10584819b4241855f5857bc1"
I20260619 14:49:36.669863 2383 raft_consensus.cc:3037] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:36.669924 2522 raft_consensus.cc:3037] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:36.672254 2522 raft_consensus.cc:2454] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9ff6c8224411a3a8806317a5c909 in term 1.
I20260619 14:49:36.672736 2174 leader_election.cc:302] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9da3df10584819b4241855f5857bc1, 5a4b9ff6c8224411a3a8806317a5c909; no voters:
I20260619 14:49:36.673066 2599 raft_consensus.cc:2781] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:36.673588 2599 raft_consensus.cc:686] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9ff6c8224411a3a8806317a5c909, State: Running, Role: LEADER
I20260619 14:49:36.674136 2599 consensus_queue.cc:237] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.674472 2383 raft_consensus.cc:2454] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9ff6c8224411a3a8806317a5c909 in term 1.
I20260619 14:49:36.679242 2086 catalog_manager.cc:5495] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9ff6c8224411a3a8806317a5c909 (127.0.241.1). New cstate: current_term: 1 leader_uuid: "5a4b9ff6c8224411a3a8806317a5c909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } health_report { overall_health: UNKNOWN } } }
W20260619 14:49:36.695070 2573 tablet.cc:2369] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:49:36.704902 2434 tablet.cc:2369] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:49:36.708978 2295 tablet.cc:2369] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:49:36.730435 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.725092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.0.241.1:33495)
0619 14:49:36.725293 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:36.725297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:36.725314 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:49:36.725610 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:36.725614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:36.725705 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:36.725837 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:36.726217 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.726904 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.728306 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.728725 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.728815 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.729269 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:36.729821 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:36.729924 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:36.730060 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 14:49:36.730122 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:49:36.738663 2383 raft_consensus.cc:1261] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9ff6c8224411a3a8806317a5c909: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:36.738729 2522 raft_consensus.cc:1261] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9ff6c8224411a3a8806317a5c909: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:36.739712 2599 consensus_queue.cc:1035] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:36.739986 2602 consensus_queue.cc:1035] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:36.751061 2605 mvcc.cc:205] Tried to move back new op lower bound from 7298582842311905280 to 7298582842062585856. Current Snapshot: MvccSnapshot[applied={T|T < 7298582842311905280}]
I20260619 14:49:36.773706 2087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:36.773988 2087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:36.777590 2087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3: Sending DeleteTablet for 3 replicas of tablet e34cf0b1d58c4a0e8e716809addf6270
I20260619 14:49:36.778746 2363 tablet_service.cc:1514] Processing DeleteTablet for tablet e34cf0b1d58c4a0e8e716809addf6270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:36 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:56060
I20260619 14:49:36.778836 2224 tablet_service.cc:1514] Processing DeleteTablet for tablet e34cf0b1d58c4a0e8e716809addf6270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:36 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:36810
I20260619 14:49:36.779851 2502 tablet_service.cc:1514] Processing DeleteTablet for tablet e34cf0b1d58c4a0e8e716809addf6270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:36 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:59722
I20260619 14:49:36.780735 2087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:49:36.781469 2087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:36.789916 2615 tablet_replica.cc:330] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909: stopping tablet replica
I20260619 14:49:36.790520 2615 raft_consensus.cc:2229] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:36.791198 2615 raft_consensus.cc:2258] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:36.792156 2616 tablet_replica.cc:330] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909: stopping tablet replica
I20260619 14:49:36.792517 2363 tablet_service.cc:1467] Processing CreateTablet for tablet 2f8a2fc090e24ab58d31864d507091ee (DEFAULT_TABLE table=test-table [id=d828d67f8ff74d4eb5154d2dc0870849]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:36.792608 2616 raft_consensus.cc:2229] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:49:36.792667 2223 tablet_service.cc:1467] Processing CreateTablet for tablet 2f8a2fc090e24ab58d31864d507091ee (DEFAULT_TABLE table=test-table [id=d828d67f8ff74d4eb5154d2dc0870849]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:36.792958 2363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f8a2fc090e24ab58d31864d507091ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:36.793294 2223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f8a2fc090e24ab58d31864d507091ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:36.793394 2615 ts_tablet_manager.cc:1905] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:36.793391 2616 raft_consensus.cc:2258] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:36.795996 2616 ts_tablet_manager.cc:1905] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:36.796831 2501 tablet_service.cc:1467] Processing CreateTablet for tablet 2f8a2fc090e24ab58d31864d507091ee (DEFAULT_TABLE table=test-table [id=d828d67f8ff74d4eb5154d2dc0870849]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:36.797869 2501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f8a2fc090e24ab58d31864d507091ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:36.800551 2592 tablet_bootstrap.cc:492] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909: Bootstrap starting.
I20260619 14:49:36.801155 2591 tablet_bootstrap.cc:492] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909: Bootstrap starting.
I20260619 14:49:36.802470 2592 tablet_bootstrap.cc:654] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:36.802735 2591 tablet_bootstrap.cc:654] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:36.803251 2617 tablet_replica.cc:330] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1: stopping tablet replica
I20260619 14:49:36.803812 2617 raft_consensus.cc:2229] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:36.804934 2617 raft_consensus.cc:2258] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:36.806516 2593 tablet_bootstrap.cc:492] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1: Bootstrap starting.
I20260619 14:49:36.808014 2593 tablet_bootstrap.cc:654] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:36.808014 2591 tablet_bootstrap.cc:492] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909: No bootstrap required, opened a new log
I20260619 14:49:36.808207 2591 ts_tablet_manager.cc:1397] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:49:36.809036 2591 raft_consensus.cc:348] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.809186 2617 ts_tablet_manager.cc:1905] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:36.809258 2591 raft_consensus.cc:374] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:36.809343 2591 raft_consensus.cc:729] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763ccb8f1fa7443cba7124d1697fb909, State: Initialized, Role: FOLLOWER
I20260619 14:49:36.809638 2591 consensus_queue.cc:260] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.811326 2593 tablet_bootstrap.cc:492] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1: No bootstrap required, opened a new log
I20260619 14:49:36.811506 2593 ts_tablet_manager.cc:1397] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:49:36.812331 2593 raft_consensus.cc:348] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.812551 2593 raft_consensus.cc:374] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:36.812655 2593 raft_consensus.cc:729] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9da3df10584819b4241855f5857bc1, State: Initialized, Role: FOLLOWER
I20260619 14:49:36.812353 2591 ts_tablet_manager.cc:1428] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:49:36.812973 2593 consensus_queue.cc:260] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.813556 2593 ts_tablet_manager.cc:1428] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:49:36.818687 2592 tablet_bootstrap.cc:492] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909: No bootstrap required, opened a new log
I20260619 14:49:36.818867 2592 ts_tablet_manager.cc:1397] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20260619 14:49:36.819625 2592 raft_consensus.cc:348] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.819797 2592 raft_consensus.cc:374] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:36.819954 2592 raft_consensus.cc:729] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9ff6c8224411a3a8806317a5c909, State: Initialized, Role: FOLLOWER
I20260619 14:49:36.820227 2592 consensus_queue.cc:260] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.820794 2592 ts_tablet_manager.cc:1428] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:49:36.878170 2615 ts_tablet_manager.cc:1918] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:49:36.878382 2615 log.cc:1196] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/e34cf0b1d58c4a0e8e716809addf6270
I20260619 14:49:36.878955 2615 ts_tablet_manager.cc:1939] T e34cf0b1d58c4a0e8e716809addf6270 P 763ccb8f1fa7443cba7124d1697fb909: Deleting consensus metadata
I20260619 14:49:36.879973 2597 raft_consensus.cc:482] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:36.880175 2597 raft_consensus.cc:504] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.882055 2597 leader_election.cc:288] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4b9ff6c8224411a3a8806317a5c909 (127.0.241.1:33495), 3c9da3df10584819b4241855f5857bc1 (127.0.241.3:43181)
I20260619 14:49:36.889847 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.883133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:37521 (local address 127.0.241.3:43181)
0619 14:49:36.883337 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:36.883346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:36.884315 (+ 969us) server_negotiation.cc:407] Connection header received
0619 14:49:36.884575 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:36.884582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:36.884665 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:36.884766 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:36.885325 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.886181 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.888072 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.888880 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.889007 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.889102 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:36.889436 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 14:49:36.889577 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:49:36.890231 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.883040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:43181 (local address 127.0.241.2:37521)
0619 14:49:36.884202 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:36.884217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:36.884425 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:36.884790 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:36.884794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:36.884816 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:36.885164 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.885171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.886311 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.886318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.887928 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.887939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.889073 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.889080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.889256 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.889295 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:49:36.889393 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:49:36.889535 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":15,"thread_start_us":522,"threads_started":1}
I20260619 14:49:36.890854 2522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f8a2fc090e24ab58d31864d507091ee" candidate_uuid: "763ccb8f1fa7443cba7124d1697fb909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9da3df10584819b4241855f5857bc1" is_pre_election: true
I20260619 14:49:36.891088 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.882312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33495 (local address 127.0.241.2:34121)
0619 14:49:36.882925 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:36.882946 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:36.883205 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:36.883770 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:36.883775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:36.884214 (+ 439us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:36.884636 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.884649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.886238 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.886243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.888600 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:36.888616 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.890234 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.890238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:36.890450 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.890485 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:49:36.890567 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:49:36.890692 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":70,"thread_start_us":316,"threads_started":1}
I20260619 14:49:36.891057 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.882382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:34121 (local address 127.0.241.1:33495)
0619 14:49:36.882650 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:36.882657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:36.883032 (+ 375us) server_negotiation.cc:407] Connection header received
0619 14:49:36.883372 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:36.883379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:36.883505 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:36.883572 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:36.884820 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.886058 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.888763 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:36.889867 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:36.890027 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:36.890127 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:36.890603 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 14:49:36.890731 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:49:36.891156 2522 raft_consensus.cc:2454] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763ccb8f1fa7443cba7124d1697fb909 in term 0.
I20260619 14:49:36.891790 2314 leader_election.cc:302] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9da3df10584819b4241855f5857bc1, 763ccb8f1fa7443cba7124d1697fb909; no voters:
I20260619 14:49:36.892212 2597 raft_consensus.cc:2781] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:36.892324 2597 raft_consensus.cc:482] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:36.892432 2597 raft_consensus.cc:3037] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:36.892829 2249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f8a2fc090e24ab58d31864d507091ee" candidate_uuid: "763ccb8f1fa7443cba7124d1697fb909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9ff6c8224411a3a8806317a5c909" is_pre_election: true
I20260619 14:49:36.893371 2249 raft_consensus.cc:2454] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763ccb8f1fa7443cba7124d1697fb909 in term 0.
I20260619 14:49:36.931269 2071 catalog_manager.cc:4841] TS 763ccb8f1fa7443cba7124d1697fb909 (127.0.241.2:40943): tablet e34cf0b1d58c4a0e8e716809addf6270 (table test-table [id=3fb10f4986fe4017882f0bdbf95ab3e6]) successfully deleted
I20260619 14:49:36.976555 2617 ts_tablet_manager.cc:1918] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:49:36.976787 2617 log.cc:1196] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/e34cf0b1d58c4a0e8e716809addf6270
I20260619 14:49:36.977351 2617 ts_tablet_manager.cc:1939] T e34cf0b1d58c4a0e8e716809addf6270 P 3c9da3df10584819b4241855f5857bc1: Deleting consensus metadata
I20260619 14:49:36.977996 2616 ts_tablet_manager.cc:1918] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:49:36.978209 2616 log.cc:1196] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/e34cf0b1d58c4a0e8e716809addf6270
I20260619 14:49:36.978739 2616 ts_tablet_manager.cc:1939] T e34cf0b1d58c4a0e8e716809addf6270 P 5a4b9ff6c8224411a3a8806317a5c909: Deleting consensus metadata
I20260619 14:49:36.979327 2597 raft_consensus.cc:504] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:36.980697 2597 leader_election.cc:288] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [CANDIDATE]: Term 1 election: Requested vote from peers 5a4b9ff6c8224411a3a8806317a5c909 (127.0.241.1:33495), 3c9da3df10584819b4241855f5857bc1 (127.0.241.3:43181)
I20260619 14:49:36.980808 2249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f8a2fc090e24ab58d31864d507091ee" candidate_uuid: "763ccb8f1fa7443cba7124d1697fb909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4b9ff6c8224411a3a8806317a5c909"
I20260619 14:49:36.981029 2249 raft_consensus.cc:3037] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:36.981348 2522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f8a2fc090e24ab58d31864d507091ee" candidate_uuid: "763ccb8f1fa7443cba7124d1697fb909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9da3df10584819b4241855f5857bc1"
I20260619 14:49:36.981550 2522 raft_consensus.cc:3037] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:36.985872 2071 catalog_manager.cc:4841] TS 3c9da3df10584819b4241855f5857bc1 (127.0.241.3:43181): tablet e34cf0b1d58c4a0e8e716809addf6270 (table test-table [id=3fb10f4986fe4017882f0bdbf95ab3e6]) successfully deleted
I20260619 14:49:37.014575 2071 catalog_manager.cc:4841] TS 5a4b9ff6c8224411a3a8806317a5c909 (127.0.241.1:33495): tablet e34cf0b1d58c4a0e8e716809addf6270 (table test-table [id=3fb10f4986fe4017882f0bdbf95ab3e6]) successfully deleted
I20260619 14:49:37.072908 2249 raft_consensus.cc:2454] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 763ccb8f1fa7443cba7124d1697fb909 in term 1.
I20260619 14:49:37.072921 2522 raft_consensus.cc:2454] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 763ccb8f1fa7443cba7124d1697fb909 in term 1.
I20260619 14:49:37.073650 2314 leader_election.cc:302] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9da3df10584819b4241855f5857bc1, 763ccb8f1fa7443cba7124d1697fb909; no voters:
I20260619 14:49:37.074069 2597 raft_consensus.cc:2781] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:37.075874 2597 raft_consensus.cc:686] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 1 LEADER]: Becoming Leader. State: Replica: 763ccb8f1fa7443cba7124d1697fb909, State: Running, Role: LEADER
I20260619 14:49:37.076666 2597 consensus_queue.cc:237] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:37.081346 2086 catalog_manager.cc:5495] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 reported cstate change: term changed from 0 to 1, leader changed from <none> to 763ccb8f1fa7443cba7124d1697fb909 (127.0.241.2). New cstate: current_term: 1 leader_uuid: "763ccb8f1fa7443cba7124d1697fb909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:49:37 dist-test-slave-p3ml krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:37.149078 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.133949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.0.241.62:33137)
0619 14:49:37.135550 (+ 1601us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:37.135558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:37.135582 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:49:37.135633 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:37.135642 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:37.135691 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:37.135804 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:37.137064 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.137999 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.139351 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.139826 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.139982 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.143845 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:37.143850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:37.143853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:37.143878 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:37.146409 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:37.147035 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:37.147043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:37.147047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:37.147153 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:37.147559 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:37.147565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:37.147570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:37.147813 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:37.147978 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:37.148396 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 14:49:37.148698 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":1458,"thread_start_us":1324,"threads_started":1}
I20260619 14:49:37.149084 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.133907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33137 (local address 127.0.0.1:57004)
0619 14:49:37.135138 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:37.135172 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:37.135507 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:37.135830 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:37.135835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:37.136734 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:37.136909 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:37.136918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.138128 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.138134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:37.139222 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:37.139234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.140006 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.140012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:37.140176 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.140869 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:37.140886 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:37.143645 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:37.146533 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:37.146546 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:37.146554 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:37.146925 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:37.147254 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:37.147261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:37.147265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:37.147473 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:37.147961 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:37.147969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:37.148125 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 14:49:37.148292 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 14:49:37.148582 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1081,"thread_start_us":950,"threads_started":1}
I20260619 14:49:37.154922 2086 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:57004:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:49:37.155997 2086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:37.162912 2224 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7f4f3267b74af4b5f69fdddea5bb87 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=121e583f16bc4162b23c1c3a1381608d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:49:37.163579 2224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7f4f3267b74af4b5f69fdddea5bb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:37.163514 2363 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7f4f3267b74af4b5f69fdddea5bb87 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=121e583f16bc4162b23c1c3a1381608d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:49:37.164018 2363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7f4f3267b74af4b5f69fdddea5bb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:37.164505 2501 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7f4f3267b74af4b5f69fdddea5bb87 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=121e583f16bc4162b23c1c3a1381608d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:49:37.165398 2501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7f4f3267b74af4b5f69fdddea5bb87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:37.171308 2592 tablet_bootstrap.cc:492] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: Bootstrap starting.
I20260619 14:49:37.172497 2591 tablet_bootstrap.cc:492] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909: Bootstrap starting.
I20260619 14:49:37.173264 2592 tablet_bootstrap.cc:654] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:37.174129 2591 tablet_bootstrap.cc:654] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:37.176868 2591 tablet_bootstrap.cc:492] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909: No bootstrap required, opened a new log
I20260619 14:49:37.177106 2591 ts_tablet_manager.cc:1397] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:49:37.177999 2591 raft_consensus.cc:348] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:37.178359 2591 raft_consensus.cc:374] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:37.178488 2591 raft_consensus.cc:729] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763ccb8f1fa7443cba7124d1697fb909, State: Initialized, Role: FOLLOWER
I20260619 14:49:37.178823 2591 consensus_queue.cc:260] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:37.179592 2597 tablet_replica.cc:439] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } } }
I20260619 14:49:37.179878 2597 tablet_replica.cc:442] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:37.180927 2591 ts_tablet_manager.cc:1428] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:49:37.176796 2593 tablet_bootstrap.cc:492] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1: Bootstrap starting.
I20260619 14:49:37.182610 2593 tablet_bootstrap.cc:654] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:37.185384 2593 tablet_bootstrap.cc:492] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1: No bootstrap required, opened a new log
I20260619 14:49:37.185616 2593 ts_tablet_manager.cc:1397] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260619 14:49:37.185945 2592 tablet_bootstrap.cc:492] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: No bootstrap required, opened a new log
I20260619 14:49:37.186141 2592 ts_tablet_manager.cc:1397] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20260619 14:49:37.186403 2593 raft_consensus.cc:348] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:37.186631 2593 raft_consensus.cc:374] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:37.186708 2593 raft_consensus.cc:729] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9da3df10584819b4241855f5857bc1, State: Initialized, Role: FOLLOWER
I20260619 14:49:37.186870 2592 raft_consensus.cc:348] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:37.187124 2592 raft_consensus.cc:374] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:37.187194 2592 raft_consensus.cc:729] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4b9ff6c8224411a3a8806317a5c909, State: Initialized, Role: FOLLOWER
I20260619 14:49:37.187356 2593 consensus_queue.cc:260] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:37.187494 2592 consensus_queue.cc:260] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:37.187970 2593 ts_tablet_manager.cc:1428] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:49:37.187863 2598 tablet_replica.cc:439] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } } }
I20260619 14:49:37.188189 2598 tablet_replica.cc:442] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:37.188939 2612 tablet_replica.cc:439] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } } }
I20260619 14:49:37.189275 2612 tablet_replica.cc:442] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:37.191650 2592 ts_tablet_manager.cc:1428] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260619 14:49:37.443279 2612 raft_consensus.cc:482] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:37.443504 2612 raft_consensus.cc:504] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:37.444150 2612 leader_election.cc:288] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 763ccb8f1fa7443cba7124d1697fb909 (127.0.241.2:40943), 3c9da3df10584819b4241855f5857bc1 (127.0.241.3:43181)
I20260619 14:49:37.444695 2383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7f4f3267b74af4b5f69fdddea5bb87" candidate_uuid: "5a4b9ff6c8224411a3a8806317a5c909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763ccb8f1fa7443cba7124d1697fb909" is_pre_election: true
I20260619 14:49:37.444758 2522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7f4f3267b74af4b5f69fdddea5bb87" candidate_uuid: "5a4b9ff6c8224411a3a8806317a5c909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9da3df10584819b4241855f5857bc1" is_pre_election: true
I20260619 14:49:37.444919 2383 raft_consensus.cc:2454] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9ff6c8224411a3a8806317a5c909 in term 0.
I20260619 14:49:37.445008 2522 raft_consensus.cc:2454] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4b9ff6c8224411a3a8806317a5c909 in term 0.
I20260619 14:49:37.445344 2174 leader_election.cc:302] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9ff6c8224411a3a8806317a5c909, 763ccb8f1fa7443cba7124d1697fb909; no voters:
I20260619 14:49:37.445614 2612 raft_consensus.cc:2781] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:37.445737 2612 raft_consensus.cc:482] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:37.445847 2612 raft_consensus.cc:3037] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:37.474840 2597 consensus_queue.cc:1035] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:37.481643 2597 consensus_queue.cc:1035] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:37.484400 2612 raft_consensus.cc:504] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:37.485076 2612 leader_election.cc:288] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [CANDIDATE]: Term 1 election: Requested vote from peers 763ccb8f1fa7443cba7124d1697fb909 (127.0.241.2:40943), 3c9da3df10584819b4241855f5857bc1 (127.0.241.3:43181)
I20260619 14:49:37.485728 2383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7f4f3267b74af4b5f69fdddea5bb87" candidate_uuid: "5a4b9ff6c8224411a3a8806317a5c909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763ccb8f1fa7443cba7124d1697fb909"
I20260619 14:49:37.485922 2383 raft_consensus.cc:3037] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:37.488623 2383 raft_consensus.cc:2454] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9ff6c8224411a3a8806317a5c909 in term 1.
I20260619 14:49:37.488979 2174 leader_election.cc:302] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4b9ff6c8224411a3a8806317a5c909, 763ccb8f1fa7443cba7124d1697fb909; no voters:
I20260619 14:49:37.488967 2521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7f4f3267b74af4b5f69fdddea5bb87" candidate_uuid: "5a4b9ff6c8224411a3a8806317a5c909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9da3df10584819b4241855f5857bc1"
I20260619 14:49:37.489183 2521 raft_consensus.cc:3037] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:37.489279 2612 raft_consensus.cc:2781] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:37.489428 2612 raft_consensus.cc:686] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [term 1 LEADER]: Becoming Leader. State: Replica: 5a4b9ff6c8224411a3a8806317a5c909, State: Running, Role: LEADER
I20260619 14:49:37.489678 2612 consensus_queue.cc:237] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } }
I20260619 14:49:37.491673 2635 tablet_replica.cc:439] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: TxnStatusTablet state changed. Reason: New leader 5a4b9ff6c8224411a3a8806317a5c909. Latest consensus state: current_term: 1 leader_uuid: "5a4b9ff6c8224411a3a8806317a5c909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } } }
I20260619 14:49:37.491915 2635 tablet_replica.cc:442] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:49:37.492595 2643 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:49:37.492816 2643 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:49:37.492378 2085 catalog_manager.cc:5495] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4b9ff6c8224411a3a8806317a5c909 (127.0.241.1). New cstate: current_term: 1 leader_uuid: "5a4b9ff6c8224411a3a8806317a5c909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } health_report { overall_health: UNKNOWN } } }
I20260619 14:49:37.548087 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.541923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33495 (local address 127.0.0.1:60840)
0619 14:49:37.542185 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:37.542207 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:37.542382 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:37.543320 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:37.543325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:37.543340 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:49:37.543543 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:37.543556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.544729 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.544734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:37.545744 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:37.545756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.546531 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.546536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:37.546721 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.546776 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:49:37.547502 (+ 726us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:49:37.547524 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:49:37.547625 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 14:49:37.547771 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 14:49:37.548161 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.541999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.0.241.1:33495)
0619 14:49:37.543027 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:37.543032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:37.543047 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:49:37.543100 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:37.543103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:37.543167 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:37.543245 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:37.543720 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.544580 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.545907 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.546369 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.546524 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.547247 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:37.547349 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:37.547513 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:37.547660 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 14:49:37.547740 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":725,"threads_started":1}
I20260619 14:49:37.552013 2383 raft_consensus.cc:1261] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9ff6c8224411a3a8806317a5c909: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:37.552635 2635 consensus_queue.cc:1035] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:37.554739 2597 tablet_replica.cc:439] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909: TxnStatusTablet state changed. Reason: New leader 5a4b9ff6c8224411a3a8806317a5c909. Latest consensus state: current_term: 1 leader_uuid: "5a4b9ff6c8224411a3a8806317a5c909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } } }
I20260619 14:49:37.555047 2597 tablet_replica.cc:442] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:37.557758 2612 tablet_replica.cc:439] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a4b9ff6c8224411a3a8806317a5c909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } } }
I20260619 14:49:37.558073 2612 tablet_replica.cc:442] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:49:37.561321 2597 tablet_replica.cc:439] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a4b9ff6c8224411a3a8806317a5c909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } } }
I20260619 14:49:37.561585 2597 tablet_replica.cc:442] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 763ccb8f1fa7443cba7124d1697fb909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:37.585975 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.565299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.0.241.2:40943)
0619 14:49:37.579917 (+ 14618us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:37.579922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:37.579947 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:49:37.580036 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:37.580041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:37.580134 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:37.580342 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:37.580841 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.581611 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.583215 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.583678 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.583908 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.584439 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:37.585202 (+ 763us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:37.585366 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:37.585504 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 14:49:37.585588 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14483,"thread_start_us":14331,"threads_started":1}
I20260619 14:49:37.586275 2521 raft_consensus.cc:2454] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4b9ff6c8224411a3a8806317a5c909 in term 1.
I20260619 14:49:37.587726 2522 raft_consensus.cc:1261] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1 [term 1 FOLLOWER]: Refusing update from remote peer 5a4b9ff6c8224411a3a8806317a5c909: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:37.591167 2635 consensus_queue.cc:1035] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:37.608249 2598 tablet_replica.cc:439] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1: TxnStatusTablet state changed. Reason: New leader 5a4b9ff6c8224411a3a8806317a5c909. Latest consensus state: current_term: 1 leader_uuid: "5a4b9ff6c8224411a3a8806317a5c909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } } }
I20260619 14:49:37.608559 2598 tablet_replica.cc:442] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:37.609045 2655 tablet_replica.cc:439] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a4b9ff6c8224411a3a8806317a5c909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } } }
I20260619 14:49:37.609365 2655 tablet_replica.cc:442] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 3c9da3df10584819b4241855f5857bc1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:37.609760 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.600503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:55053 (local address 127.0.241.1:33495)
0619 14:49:37.600766 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:37.600775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:37.602474 (+ 1699us) server_negotiation.cc:407] Connection header received
0619 14:49:37.602694 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:37.602701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:37.602796 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:37.602899 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:37.603550 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.604669 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.607489 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.608095 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.608223 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.608780 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:37.608928 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:37.609068 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:37.609263 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 14:49:37.609492 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:49:37.610448 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.599555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33495 (local address 127.0.241.2:55053)
0619 14:49:37.602339 (+ 2784us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:37.602364 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:37.602551 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:37.602941 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:37.602948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:37.602967 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:49:37.603346 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:37.603359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.604940 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.604948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:37.607295 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:37.607314 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.608270 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.608277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:37.608460 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.608540 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:49:37.609122 (+ 582us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:49:37.609141 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:49:37.609931 (+ 790us) client_negotiation.cc:240] Negotiation successful
0619 14:49:37.610076 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2567,"thread_start_us":2374,"threads_started":1}
I20260619 14:49:37.614488 2659 tablet_replica.cc:439] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a4b9ff6c8224411a3a8806317a5c909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4b9ff6c8224411a3a8806317a5c909" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33495 } } peers { permanent_uuid: "763ccb8f1fa7443cba7124d1697fb909" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40943 } } peers { permanent_uuid: "3c9da3df10584819b4241855f5857bc1" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 43181 } } }
I20260619 14:49:37.617395 2659 tablet_replica.cc:442] T 0c7f4f3267b74af4b5f69fdddea5bb87 P 5a4b9ff6c8224411a3a8806317a5c909: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:49:37.661791 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.647084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:40125 (local address 127.0.241.2:40943)
0619 14:49:37.647267 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:37.647273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:37.656131 (+ 8858us) server_negotiation.cc:407] Connection header received
0619 14:49:37.656390 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:37.656396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:37.656472 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:37.656566 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:37.657130 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.657854 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.659576 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.660573 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.660701 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.661216 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:37.661327 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:37.661399 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:37.661544 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 14:49:37.661602 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:49:37.662444 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.647043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:40943 (local address 127.0.241.1:40125)
0619 14:49:37.656035 (+ 8992us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:37.656058 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:37.656249 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:37.656578 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:37.656583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:37.656594 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:49:37.656954 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:37.656962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.657981 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.657988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:37.659408 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:37.659474 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.660770 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.660778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:37.660978 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.661047 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:49:37.661433 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:49:37.661444 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 14:49:37.661533 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:49:37.661673 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8747,"thread_start_us":8574,"threads_started":1}
I20260619 14:49:37.700570 2085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:37.700821 2085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:37.703580 2085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88feb61578ad4befae45a8ee963b10e3: Sending DeleteTablet for 3 replicas of tablet 2f8a2fc090e24ab58d31864d507091ee
I20260619 14:49:37.704449 2501 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f8a2fc090e24ab58d31864d507091ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:37 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:59722
I20260619 14:49:37.704437 2363 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f8a2fc090e24ab58d31864d507091ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:37 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:56060
I20260619 14:49:37.705121 2224 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f8a2fc090e24ab58d31864d507091ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:37 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:36810
I20260619 14:49:37.725126 2667 tablet_replica.cc:330] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909: stopping tablet replica
I20260619 14:49:37.725381 2667 raft_consensus.cc:2229] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:37.725677 2667 raft_consensus.cc:2258] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:37.726009 2666 tablet_replica.cc:330] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1: stopping tablet replica
I20260619 14:49:37.726301 2666 raft_consensus.cc:2229] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:37.726610 2666 raft_consensus.cc:2258] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:37.728114 2666 ts_tablet_manager.cc:1905] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:37.730074 2667 ts_tablet_manager.cc:1905] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:37.735590 2665 tablet_replica.cc:330] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909: stopping tablet replica
I20260619 14:49:37.735775 2665 raft_consensus.cc:2229] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:49:37.736395 2665 raft_consensus.cc:2258] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:37.737407 2665 ts_tablet_manager.cc:1905] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 14:49:37 dist-test-slave-p3ml krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880576, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:49:37.750415 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.727114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.0.241.1:33495)
0619 14:49:37.727302 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:37.727311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:37.732506 (+ 5195us) server_negotiation.cc:407] Connection header received
0619 14:49:37.732790 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:37.732796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:37.732910 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:37.732982 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:37.735328 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.736444 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.738549 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.739264 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.739366 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.743969 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:37.743992 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:37.744005 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:37.744040 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:37.747646 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:37.748299 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:37.748308 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:37.748312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:37.748445 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:37.748822 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:37.748835 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:37.748838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:37.749189 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:37.749471 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:37.749805 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 14:49:37.750039 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 14:49:37.751163 2289 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60852
I20260619 14:49:37.776285 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.754155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.0.241.62:33137)
0619 14:49:37.763889 (+ 9734us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:37.763894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:37.763909 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:49:37.763957 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:37.763960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:37.764015 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:37.764134 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:37.765401 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.766222 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.767423 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.767879 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.768031 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.770746 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:37.770752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:37.770757 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:37.770795 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:37.773349 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:37.774015 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:37.774022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:37.774025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:37.774208 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:37.774616 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:37.774622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:37.774625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:37.774945 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:37.775187 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:37.775643 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 14:49:37.775868 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9617,"thread_start_us":9468,"threads_started":1}
W20260619 14:49:37.776995 2085 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51808
I20260619 14:49:37.779137 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 2158
I20260619 14:49:37.795655 2666 ts_tablet_manager.cc:1918] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:49:37.795915 2666 log.cc:1196] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/2f8a2fc090e24ab58d31864d507091ee
I20260619 14:49:37.796432 2666 ts_tablet_manager.cc:1939] T 2f8a2fc090e24ab58d31864d507091ee P 3c9da3df10584819b4241855f5857bc1: Deleting consensus metadata
I20260619 14:49:37.798705 2667 ts_tablet_manager.cc:1918] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:49:37.799270 2667 log.cc:1196] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/2f8a2fc090e24ab58d31864d507091ee
I20260619 14:49:37.800812 2667 ts_tablet_manager.cc:1939] T 2f8a2fc090e24ab58d31864d507091ee P 5a4b9ff6c8224411a3a8806317a5c909: Deleting consensus metadata
I20260619 14:49:37.809937 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.781598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.0.241.1:33495)
0619 14:49:37.781935 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:37.781944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:37.790691 (+ 8747us) server_negotiation.cc:407] Connection header received
0619 14:49:37.791091 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:37.791097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:37.791172 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:37.791280 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:37.799421 (+ 8141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.800185 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.801753 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.802177 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.802311 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.804470 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:37.804476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:37.804480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:37.804538 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:37.807216 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:37.807908 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:37.807914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:37.807917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:37.808072 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:37.808447 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:37.808451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:37.808453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:37.808704 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:37.809031 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:37.809318 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:49:37.809568 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
W20260619 14:49:37.810491 2289 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60868
W20260619 14:49:37.811058 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 2158: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:49:37.811236 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 2158
I20260619 14:49:37.860843 2071 catalog_manager.cc:4841] TS 3c9da3df10584819b4241855f5857bc1 (127.0.241.3:43181): tablet 2f8a2fc090e24ab58d31864d507091ee (table test-table [id=d828d67f8ff74d4eb5154d2dc0870849]) successfully deleted
W20260619 14:49:37.889026 2071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.241.1:33495 (error 108)
I20260619 14:49:37.889451 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 2298
I20260619 14:49:37.891085 2665 ts_tablet_manager.cc:1918] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:49:37.891240 2665 log.cc:1196] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/2f8a2fc090e24ab58d31864d507091ee
I20260619 14:49:37.891772 2665 ts_tablet_manager.cc:1939] T 2f8a2fc090e24ab58d31864d507091ee P 763ccb8f1fa7443cba7124d1697fb909: Deleting consensus metadata
I20260619 14:49:37.891968 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.890530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33495 (local address 127.0.0.1:60876)
0619 14:49:37.891125 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:37.891344 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"threads_started":1}
W20260619 14:49:37.892247 2071 catalog_manager.cc:4584] TS 5a4b9ff6c8224411a3a8806317a5c909 (127.0.241.1:33495): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f8a2fc090e24ab58d31864d507091ee: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33495: connect: Connection refused (error 111)
I20260619 14:49:37.894819 2071 catalog_manager.cc:4841] TS 763ccb8f1fa7443cba7124d1697fb909 (127.0.241.2:40943): tablet 2f8a2fc090e24ab58d31864d507091ee (table test-table [id=d828d67f8ff74d4eb5154d2dc0870849]) successfully deleted
Jun 19 14:49:37 dist-test-slave-p3ml krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880576, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:49:37.907058 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.890251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.0.241.2:40943)
0619 14:49:37.890523 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:37.890530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:37.890550 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:49:37.890760 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:37.890766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:37.890841 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:37.890981 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:37.892792 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.894006 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.895826 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.896461 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.896556 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.901079 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:37.901100 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:37.901114 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:37.901152 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:37.904268 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:37.904960 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:37.904967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:37.904970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:37.905091 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:37.905486 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:37.905490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:37.905492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:37.905836 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:37.906216 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:37.906500 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 14:49:37.906714 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
W20260619 14:49:37.907694 2428 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47416
W20260619 14:49:37.908366 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 2298: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:49:37.908537 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 2298
I20260619 14:49:37.927230 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 2437
Jun 19 14:49:37 dist-test-slave-p3ml krb5kdc[2042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880576, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:49:37.941948 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.928000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.0.241.3:43181)
0619 14:49:37.928504 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:37.928508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:37.928550 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:49:37.928615 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:37.928623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:37.928709 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:37.928832 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:37.929961 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.930770 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.932152 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:37.932619 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:37.932714 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:37.936732 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:37.936745 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:37.936759 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:37.936789 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:37.939723 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:37.940307 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:37.940314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:37.940316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:37.940441 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:37.940757 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:37.940760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:37.940762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:37.941104 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:37.941345 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:37.941545 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 14:49:37.941695 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":283,"threads_started":1}
W20260619 14:49:37.942633 2567 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47252
W20260619 14:49:37.943230 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 2437: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:49:37.943370 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 2437
I20260619 14:49:37.946105 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.945714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33495 (local address 127.0.0.1:60890)
0619 14:49:37.945891 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:37.945966 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 14:49:37.947139 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.946802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33495 (local address 127.0.0.1:60898)
0619 14:49:37.946953 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:37.947015 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 14:49:37.960073 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 2055
I20260619 14:49:37.961061 2127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:38.075632 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:38.074270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33495 (local address 127.0.0.1:60914)
0619 14:49:38.075306 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:38.075408 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":879}
I20260619 14:49:38.076512 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 2055
2026-06-19T14:49:38Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4883 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:49:40 dist-test-slave-p3ml krb5kdc[2680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:49:40 dist-test-slave-p3ml krb5kdc[2680](info): set up 2 sockets
Jun 19 14:49:40 dist-test-slave-p3ml krb5kdc[2680](info): commencing operation
krb5kdc: starting...
W20260619 14:49:41.095652 964 mini_kdc.cc:121] Time spent starting KDC: real 2.973s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:49:41 dist-test-slave-p3ml krb5kdc[2680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:49:41.278448 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:34351
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:34351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:41.596625 2693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:41.596937 2693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:41.597090 2693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:41.604095 2693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:41.604221 2693 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:49:41.604300 2693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:41.604405 2693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:41.604480 2693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:41.604557 2693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:41.613811 2693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:34351
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:34351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:41.614961 2693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:41.616276 2693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:41.627928 2698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:41.627928 2701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:41.627955 2699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:41.628743 2693 server_base.cc:1034] running on GCE node
I20260619 14:49:41.629400 2693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:41.630625 2693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:41.631855 2693 hybrid_clock.cc:648] HybridClock initialized: now 1781880581631798 us; error 56 us; skew 500 ppm
I20260619 14:49:41.632161 2693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:41 dist-test-slave-p3ml krb5kdc[2680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:41.634938 2693 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:49:41.636253 2693 webserver.cc:460] Webserver started at http://127.0.241.62:41193/ using document root <none> and password file <none>
I20260619 14:49:41.636839 2693 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:41.636919 2693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:41.637177 2693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:41.639884 2693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "7ce4f0c9b92b49febada095b584cf50e"
format_stamp: "Formatted at 2026-06-19 14:49:41 on dist-test-slave-p3ml"
server_key: "c9818a29ab99bf95434c6eeb69d359c8"
server_key_iv: "7df7a4597d80f0a6759e3dcd0693d020"
server_key_version: "encryptionkey@0"
I20260619 14:49:41.640549 2693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "7ce4f0c9b92b49febada095b584cf50e"
format_stamp: "Formatted at 2026-06-19 14:49:41 on dist-test-slave-p3ml"
server_key: "c9818a29ab99bf95434c6eeb69d359c8"
server_key_iv: "7df7a4597d80f0a6759e3dcd0693d020"
server_key_version: "encryptionkey@0"
I20260619 14:49:41.645278 2693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 14:49:41.648360 2708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:41.649544 2693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:49:41.649708 2693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "7ce4f0c9b92b49febada095b584cf50e"
format_stamp: "Formatted at 2026-06-19 14:49:41 on dist-test-slave-p3ml"
server_key: "c9818a29ab99bf95434c6eeb69d359c8"
server_key_iv: "7df7a4597d80f0a6759e3dcd0693d020"
server_key_version: "encryptionkey@0"
I20260619 14:49:41.649914 2693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:41.694669 2693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:41.698622 2693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:41.698859 2693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:41.719389 2693 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:34351
I20260619 14:49:41.719410 2769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:34351 every 8 connection(s)
I20260619 14:49:41.720683 2693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:41.724261 2770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:41.725426 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 2693
I20260619 14:49:41.725611 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:49:41.726279 964 external_mini_cluster.cc:1428] Setting key e3aba00381b395bf696644c143f973e2
I20260619 14:49:41.737929 2770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e: Bootstrap starting.
Jun 19 14:49:41 dist-test-slave-p3ml krb5kdc[2680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:41.740918 2770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:41.742173 2770 log.cc:824] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:41.744957 2770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e: No bootstrap required, opened a new log
I20260619 14:49:41.748721 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:41.728301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.0.241.62:34351)
0619 14:49:41.729152 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:41.729166 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:41.729219 (+ 53us) server_negotiation.cc:407] Connection header received
0619 14:49:41.730984 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:41.731012 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:41.731515 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:41.731907 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:41.733114 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:41.735184 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:41.736438 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:41.737453 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:41.737652 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:41.741466 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:41.741476 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:41.741494 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:41.741535 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:41.744737 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:41.745491 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:41.745497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:41.745502 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:41.745668 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:41.746051 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:41.746056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:41.746059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:41.746567 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:41.746907 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:41.747303 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 14:49:41.747750 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":371,"threads_started":1}
I20260619 14:49:41.752898 2770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce4f0c9b92b49febada095b584cf50e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34351 } }
I20260619 14:49:41.753255 2770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:41.753391 2770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce4f0c9b92b49febada095b584cf50e, State: Initialized, Role: FOLLOWER
I20260619 14:49:41.753973 2770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce4f0c9b92b49febada095b584cf50e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34351 } }
I20260619 14:49:41.754213 2770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:41.754374 2770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:41.754542 2770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:41.757164 2770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce4f0c9b92b49febada095b584cf50e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34351 } }
I20260619 14:49:41.757640 2770 leader_election.cc:302] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ce4f0c9b92b49febada095b584cf50e; no voters:
I20260619 14:49:41.758241 2770 leader_election.cc:288] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:49:41.758358 2775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:41.759212 2775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [term 1 LEADER]: Becoming Leader. State: Replica: 7ce4f0c9b92b49febada095b584cf50e, State: Running, Role: LEADER
I20260619 14:49:41.759893 2775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce4f0c9b92b49febada095b584cf50e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34351 } }
I20260619 14:49:41.760993 2770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:41.762679 2777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ce4f0c9b92b49febada095b584cf50e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce4f0c9b92b49febada095b584cf50e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34351 } } }
I20260619 14:49:41.762671 2776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ce4f0c9b92b49febada095b584cf50e. Latest consensus state: current_term: 1 leader_uuid: "7ce4f0c9b92b49febada095b584cf50e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce4f0c9b92b49febada095b584cf50e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34351 } } }
I20260619 14:49:41.763139 2777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:41.763230 2776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:41.764693 2780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:41.770335 2780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:41.778496 2780 catalog_manager.cc:1347] Generated new cluster ID: 0417fcf930614d7bbd9e37c3ec86dea0
I20260619 14:49:41.778616 2780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:41.789170 2780 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:41.790347 2780 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:41.800823 2780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e: Generated new TSK 0
I20260619 14:49:41.801578 2780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:49:41.851018 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34351
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:42.166733 2796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:42.166970 2796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:42.167150 2796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:42.174063 2796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:42.174153 2796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:42.174284 2796 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:49:42.174376 2796 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:49:42.174505 2796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:49:42.183792 2796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.241.62:34351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:42.184957 2796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:42.186265 2796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:42.198797 2801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:42.198740 2804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:42.198828 2802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:42.199827 2796 server_base.cc:1034] running on GCE node
I20260619 14:49:42.200410 2796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:42.201375 2796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:42.202602 2796 hybrid_clock.cc:648] HybridClock initialized: now 1781880582202571 us; error 45 us; skew 500 ppm
I20260619 14:49:42.202907 2796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:42 dist-test-slave-p3ml krb5kdc[2680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:42.206089 2796 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:49:42.207666 2796 webserver.cc:460] Webserver started at http://127.0.241.1:36525/ using document root <none> and password file <none>
I20260619 14:49:42.208354 2796 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:42.208557 2796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:42.208866 2796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:42.211786 2796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "c140e639feee4ea9a7daf027f7d36e52"
format_stamp: "Formatted at 2026-06-19 14:49:42 on dist-test-slave-p3ml"
server_key: "4669fac0ff13265328307ee8b8a5601d"
server_key_iv: "4de8fd4fede7b0c4e393bf9051af3433"
server_key_version: "encryptionkey@0"
I20260619 14:49:42.212610 2796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "c140e639feee4ea9a7daf027f7d36e52"
format_stamp: "Formatted at 2026-06-19 14:49:42 on dist-test-slave-p3ml"
server_key: "4669fac0ff13265328307ee8b8a5601d"
server_key_iv: "4de8fd4fede7b0c4e393bf9051af3433"
server_key_version: "encryptionkey@0"
I20260619 14:49:42.217335 2796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:49:42.220489 2811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:42.221701 2796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:49:42.221874 2796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "c140e639feee4ea9a7daf027f7d36e52"
format_stamp: "Formatted at 2026-06-19 14:49:42 on dist-test-slave-p3ml"
server_key: "4669fac0ff13265328307ee8b8a5601d"
server_key_iv: "4de8fd4fede7b0c4e393bf9051af3433"
server_key_version: "encryptionkey@0"
I20260619 14:49:42.222064 2796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:42.246436 2796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:42.250362 2796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:42.250658 2796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:42.253417 2796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:42.253542 2796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:42.253674 2796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:42.253772 2796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:49:42 dist-test-slave-p3ml krb5kdc[2680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:42.276865 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:42.253998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:38445 (local address 127.0.241.62:34351)
0619 14:49:42.254702 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:42.254708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:42.255807 (+ 1099us) server_negotiation.cc:407] Connection header received
0619 14:49:42.257874 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:42.257879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:42.257963 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:42.258073 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:42.260456 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.261357 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.263060 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.263998 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.264137 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:42.269647 (+ 5510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:42.269655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:42.269660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:42.269706 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:42.272716 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:42.273799 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:42.273803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:42.273806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:42.273934 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:42.274601 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:42.274605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:42.274607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:42.274870 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:42.275003 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:42.276143 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0619 14:49:42.276417 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":482,"threads_started":1}
I20260619 14:49:42.278878 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:42.254747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34351 (local address 127.0.241.1:38445)
0619 14:49:42.255634 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:42.255678 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:42.257603 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:42.258519 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:42.258527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:42.259305 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:42.260272 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:42.260297 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.261568 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.261576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:42.262914 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:42.262925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.264461 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.264469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:42.264716 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:42.265616 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:42.265672 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:42.269369 (+ 3697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:42.272855 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:42.272868 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:42.272891 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:42.273662 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:42.274046 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:42.274054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:42.274061 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:42.274373 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:42.275007 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:42.275018 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:42.275622 (+ 604us) client_negotiation.cc:769] Sending connection context
0619 14:49:42.276014 (+ 392us) client_negotiation.cc:240] Negotiation successful
0619 14:49:42.276706 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":375,"threads_started":1}
I20260619 14:49:42.299222 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:42.284919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:39781 (local address 127.0.241.62:34351)
0619 14:49:42.285093 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:42.285099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:42.285859 (+ 760us) server_negotiation.cc:407] Connection header received
0619 14:49:42.286053 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:42.286059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:42.286161 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:42.286276 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:42.287256 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.288507 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.290611 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.291056 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.291182 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:42.293647 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:42.293655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:42.293661 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:42.293703 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:42.296667 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:42.297347 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:42.297353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:42.297359 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:42.297487 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:42.297824 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:42.297838 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:42.297841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:42.298105 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:42.298253 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:42.298641 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 14:49:42.298868 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:49:42.299314 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:42.284909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34351 (local address 127.0.241.1:39781)
0619 14:49:42.285709 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:42.285739 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:42.285915 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:42.286323 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:42.286341 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:42.286839 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:42.287117 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:42.287126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.288686 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.288693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:42.290469 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:42.290481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.291242 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.291249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:42.291426 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:42.292173 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:42.292203 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:42.293402 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:42.296799 (+ 3397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:42.296805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:42.296810 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:42.297238 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:42.297592 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:42.297598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:42.297601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:42.297724 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:42.298229 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:42.298234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:42.298415 (+ 181us) client_negotiation.cc:769] Sending connection context
0619 14:49:42.298554 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 14:49:42.298876 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":503,"threads_started":1}
I20260619 14:49:42.307889 2796 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:36757
I20260619 14:49:42.307929 2931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:36757 every 8 connection(s)
I20260619 14:49:42.309067 2796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:42.310911 2932 heartbeater.cc:347] Connected to a master server at 127.0.241.62:34351
I20260619 14:49:42.311131 2932 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:42.311774 2932 heartbeater.cc:511] Master 127.0.241.62:34351 requested a full tablet report, sending...
I20260619 14:49:42.313575 2725 ts_manager.cc:194] Registered new tserver with Master: c140e639feee4ea9a7daf027f7d36e52 (127.0.241.1:36757)
I20260619 14:49:42.314966 2725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:38445
I20260619 14:49:42.317607 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 2796
I20260619 14:49:42.317761 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:49:42.318311 964 external_mini_cluster.cc:1428] Setting key 6c43d0ead5390c79021a54c2928f4a37
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:49:42.356431 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34351
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:42.666000 2936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:42.666246 2936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:42.666451 2936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:42.673462 2936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:42.673566 2936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:42.673699 2936 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:49:42.673758 2936 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:49:42.673902 2936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:49:42.683979 2936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.241.62:34351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:42.685205 2936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:42.686480 2936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:42.699031 2942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:42.699101 2944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:42.699067 2941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:42.700304 2936 server_base.cc:1034] running on GCE node
I20260619 14:49:42.700889 2936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:42.701774 2936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:42.703009 2936 hybrid_clock.cc:648] HybridClock initialized: now 1781880582702963 us; error 40 us; skew 500 ppm
I20260619 14:49:42.703311 2936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:42 dist-test-slave-p3ml krb5kdc[2680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:42.706219 2936 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:49:42.707679 2936 webserver.cc:460] Webserver started at http://127.0.241.2:41755/ using document root <none> and password file <none>
I20260619 14:49:42.708361 2936 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:42.708518 2936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:42.708827 2936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:42.711683 2936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "6cc15d47deb04ccba2237bf49c5b8a65"
format_stamp: "Formatted at 2026-06-19 14:49:42 on dist-test-slave-p3ml"
server_key: "1d9f8094fa5ec2411609c1ac454babe3"
server_key_iv: "0db660369a843c52ac84506ceb4e3c3e"
server_key_version: "encryptionkey@0"
I20260619 14:49:42.712421 2936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "6cc15d47deb04ccba2237bf49c5b8a65"
format_stamp: "Formatted at 2026-06-19 14:49:42 on dist-test-slave-p3ml"
server_key: "1d9f8094fa5ec2411609c1ac454babe3"
server_key_iv: "0db660369a843c52ac84506ceb4e3c3e"
server_key_version: "encryptionkey@0"
I20260619 14:49:42.717635 2936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:49:42.736068 2951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:42.737919 2936 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.014s
I20260619 14:49:42.738102 2936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "6cc15d47deb04ccba2237bf49c5b8a65"
format_stamp: "Formatted at 2026-06-19 14:49:42 on dist-test-slave-p3ml"
server_key: "1d9f8094fa5ec2411609c1ac454babe3"
server_key_iv: "0db660369a843c52ac84506ceb4e3c3e"
server_key_version: "encryptionkey@0"
I20260619 14:49:42.738350 2936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:42.774015 2936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:42.778057 2936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:42.778358 2936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:42.781276 2936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:42.781410 2936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:42.781570 2936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:42.781656 2936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:49:42 dist-test-slave-p3ml krb5kdc[2680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:42.806172 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:42.781415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:40769 (local address 127.0.241.62:34351)
0619 14:49:42.781647 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:42.781653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:42.783181 (+ 1528us) server_negotiation.cc:407] Connection header received
0619 14:49:42.785679 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:42.785685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:42.785797 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:42.785913 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:42.788083 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.789366 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.791808 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.792440 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.792599 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:42.797680 (+ 5081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:42.797688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:42.797696 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:42.797739 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:42.801630 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:42.803447 (+ 1817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:42.803455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:42.803462 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:42.803633 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:42.804267 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:42.804271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:42.804273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:42.804541 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:42.804739 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:42.805772 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0619 14:49:42.805927 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 14:49:42.808291 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:42.782107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34351 (local address 127.0.241.2:40769)
0619 14:49:42.783014 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:42.783060 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:42.785135 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:42.786167 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:42.786180 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:42.786981 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:42.787869 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:42.787901 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.789561 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.789568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:42.791661 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:42.791673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.792725 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.792736 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:42.793134 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:42.793969 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:42.794007 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:42.797360 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:42.802500 (+ 5140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:42.802511 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:42.802528 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:42.803279 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:42.803784 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:42.803791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:42.803797 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:42.804142 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:42.804722 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:42.804730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:42.805288 (+ 558us) client_negotiation.cc:769] Sending connection context
0619 14:49:42.805665 (+ 377us) client_negotiation.cc:240] Negotiation successful
0619 14:49:42.806341 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":399,"threads_started":1}
I20260619 14:49:42.828889 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:42.814143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:53551 (local address 127.0.241.62:34351)
0619 14:49:42.814354 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:42.814363 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:42.815008 (+ 645us) server_negotiation.cc:407] Connection header received
0619 14:49:42.815194 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:42.815212 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:42.815296 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:42.815409 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:42.816941 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.817879 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.819189 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.819652 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.819785 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:42.822037 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:42.822045 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:42.822051 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:42.822100 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:42.825486 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:42.826589 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:42.826596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:42.826602 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:42.826786 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:42.827298 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:42.827306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:42.827311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:42.827735 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:42.827855 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:42.828417 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 14:49:42.828614 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:49:42.829334 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:42.814151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34351 (local address 127.0.241.2:53551)
0619 14:49:42.814886 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:42.814911 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:42.815061 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:42.815469 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:42.815476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:42.816200 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:42.816765 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:42.816785 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.818008 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.818013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:42.819051 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:42.819062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:42.819854 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:42.819862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:42.820017 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:42.820782 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:42.820798 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:42.821759 (+ 961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:42.825662 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:42.825670 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:42.825676 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:42.826382 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:42.826957 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:42.826965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:42.826971 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:42.827148 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:42.827863 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:42.827870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:42.828070 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 14:49:42.828273 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 14:49:42.828681 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":375,"threads_started":1}
I20260619 14:49:42.832541 2936 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:34135
I20260619 14:49:42.832583 3070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:34135 every 8 connection(s)
I20260619 14:49:42.833716 2936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:49:42.835772 3071 heartbeater.cc:347] Connected to a master server at 127.0.241.62:34351
I20260619 14:49:42.836006 3071 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:42.836591 3071 heartbeater.cc:511] Master 127.0.241.62:34351 requested a full tablet report, sending...
I20260619 14:49:42.837777 2725 ts_manager.cc:194] Registered new tserver with Master: 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135)
I20260619 14:49:42.838542 2725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:40769
I20260619 14:49:42.843242 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 2936
I20260619 14:49:42.843379 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:49:42.843822 964 external_mini_cluster.cc:1428] Setting key 37b5aabed074e86b3c23eb866f6181c9
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:49:42.882396 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34351
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:43.195421 3075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:43.195781 3075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:43.195966 3075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:43.203022 3075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:43.203116 3075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:43.203197 3075 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:49:43.203302 3075 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:49:43.203393 3075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:49:43.213131 3075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.241.62:34351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:43.214376 3075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:43.215735 3075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:43.228262 3080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:43.228377 3081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:43.228423 3083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:43.229176 3075 server_base.cc:1034] running on GCE node
I20260619 14:49:43.229702 3075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:43.230546 3075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:43.231765 3075 hybrid_clock.cc:648] HybridClock initialized: now 1781880583231728 us; error 54 us; skew 500 ppm
I20260619 14:49:43.232074 3075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:43 dist-test-slave-p3ml krb5kdc[2680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:43.235057 3075 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:49:43.236479 3075 webserver.cc:460] Webserver started at http://127.0.241.3:38305/ using document root <none> and password file <none>
I20260619 14:49:43.237073 3075 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:43.237169 3075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:43.237424 3075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:43.240450 3075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "4e2940371c6c4d84a6261123a28014d9"
format_stamp: "Formatted at 2026-06-19 14:49:43 on dist-test-slave-p3ml"
server_key: "00fa2aec3ef5401cdbf0e6c5ea84171e"
server_key_iv: "08438aa9fa96bf8ce3d073f514f9d92b"
server_key_version: "encryptionkey@0"
I20260619 14:49:43.241142 3075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "4e2940371c6c4d84a6261123a28014d9"
format_stamp: "Formatted at 2026-06-19 14:49:43 on dist-test-slave-p3ml"
server_key: "00fa2aec3ef5401cdbf0e6c5ea84171e"
server_key_iv: "08438aa9fa96bf8ce3d073f514f9d92b"
server_key_version: "encryptionkey@0"
I20260619 14:49:43.246084 3075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:49:43.249253 3090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:43.250526 3075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:49:43.250659 3075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "4e2940371c6c4d84a6261123a28014d9"
format_stamp: "Formatted at 2026-06-19 14:49:43 on dist-test-slave-p3ml"
server_key: "00fa2aec3ef5401cdbf0e6c5ea84171e"
server_key_iv: "08438aa9fa96bf8ce3d073f514f9d92b"
server_key_version: "encryptionkey@0"
I20260619 14:49:43.250810 3075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:43.278180 3075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:43.282395 3075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:43.282642 3075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:43.285288 3075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:43.285398 3075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:43.285593 3075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:43.285691 3075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:49:43 dist-test-slave-p3ml krb5kdc[2680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:43.308488 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.285799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:53263 (local address 127.0.241.62:34351)
0619 14:49:43.286056 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:43.286066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:43.287520 (+ 1454us) server_negotiation.cc:407] Connection header received
0619 14:49:43.290212 (+ 2692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:43.290221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:43.290354 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:43.290478 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:43.293459 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.294883 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.297115 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.297506 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.297660 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.302738 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:43.302743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:43.302746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:43.302777 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:43.305386 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:43.306081 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:43.306087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:43.306092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:43.306219 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:43.306634 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:43.306643 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:43.306645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:43.306903 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:43.307058 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:43.307862 (+ 804us) server_negotiation.cc:299] Negotiation successful
0619 14:49:43.308088 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 14:49:43.309578 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.286455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34351 (local address 127.0.241.3:53263)
0619 14:49:43.287339 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.287387 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:43.289854 (+ 2467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:43.290994 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:43.291008 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:43.292068 (+ 1060us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:43.293197 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.293232 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.295043 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.295047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.296960 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.296975 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.297985 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.297990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.298263 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.299079 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:43.299118 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:43.302501 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:43.305521 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:43.305530 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:43.305545 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:43.305955 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:43.306329 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:43.306336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:43.306342 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:43.306514 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:43.307036 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:43.307043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:43.307362 (+ 319us) client_negotiation.cc:769] Sending connection context
0619 14:49:43.307714 (+ 352us) client_negotiation.cc:240] Negotiation successful
0619 14:49:43.308208 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":408,"threads_started":1}
I20260619 14:49:43.317534 2932 heartbeater.cc:503] Master 127.0.241.62:34351 was elected leader, sending a full tablet report...
I20260619 14:49:43.334731 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.319500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:36409 (local address 127.0.241.62:34351)
0619 14:49:43.319669 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:43.319677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:43.320426 (+ 749us) server_negotiation.cc:407] Connection header received
0619 14:49:43.320691 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:43.320697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:43.320803 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:43.320911 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:43.322404 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.323394 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.324784 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.325218 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.325390 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.327679 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:43.327685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:43.327689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:43.327725 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:43.331856 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:43.332714 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:43.332722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:43.332726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:43.332904 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:43.333344 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:43.333352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:43.333357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:43.333617 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:43.333777 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:43.334159 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 14:49:43.334376 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:49:43.334827 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.319535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34351 (local address 127.0.241.3:36409)
0619 14:49:43.320334 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.320353 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:43.320553 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:43.320952 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:43.320958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:43.321714 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:43.322265 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.322276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.323578 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.323585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.324638 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.324649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.325405 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.325411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.325601 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.326380 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:43.326399 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:43.327419 (+ 1020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:43.331990 (+ 4571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:43.331997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:43.332004 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:43.332542 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:43.333069 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:43.333073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:43.333078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:43.333223 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:43.333735 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:43.333739 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:43.333921 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 14:49:43.334066 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 14:49:43.334392 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":401,"threads_started":1}
I20260619 14:49:43.348384 3075 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:42833
I20260619 14:49:43.348428 3209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:42833 every 8 connection(s)
I20260619 14:49:43.349571 3075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:49:43.351145 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 3075
I20260619 14:49:43.351307 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:49:43.351686 3210 heartbeater.cc:347] Connected to a master server at 127.0.241.62:34351
I20260619 14:49:43.351871 964 external_mini_cluster.cc:1428] Setting key 2ad000c614df6a36f1daccefc0ae3d34
I20260619 14:49:43.351925 3210 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:43.352694 3210 heartbeater.cc:511] Master 127.0.241.62:34351 requested a full tablet report, sending...
I20260619 14:49:43.354269 2724 ts_manager.cc:194] Registered new tserver with Master: 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833)
I20260619 14:49:43.355373 2724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:53263
I20260619 14:49:43.366835 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:49:43 dist-test-slave-p3ml krb5kdc[2680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:49:43 dist-test-slave-p3ml krb5kdc[2680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:43.411306 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.398508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.0.241.62:34351)
0619 14:49:43.398730 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:43.398736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:43.398957 (+ 221us) server_negotiation.cc:407] Connection header received
0619 14:49:43.399099 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:43.399103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:43.399165 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:43.399248 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:43.400600 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.401313 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.402442 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.402838 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.402927 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.406495 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:43.406500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:43.406503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:43.406533 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:43.408819 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:43.409388 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:43.409392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:43.409394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:43.409506 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:43.409926 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:43.409929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:43.409931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:43.410172 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:43.410336 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:43.410781 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 14:49:43.411018 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:49:43.415126 2724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:49:43.421737 2724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:43.470031 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.433151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.0.241.2:34135)
0619 14:49:43.461597 (+ 28446us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:43.461604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:43.461629 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:49:43.461712 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:43.461720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:43.462037 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:43.462224 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:43.463351 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.464884 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.467813 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.468889 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.469042 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.469250 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:43.469551 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 14:49:43.469710 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28310,"thread_start_us":28158,"threads_started":1}
I20260619 14:49:43.470139 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.433026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:34135 (local address 127.0.0.1:36292)
0619 14:49:43.458931 (+ 25905us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.458987 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:43.459218 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:43.462254 (+ 3036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:43.462261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:43.462288 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:43.463108 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.463124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.465030 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.465035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.467636 (+ 2601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.467651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.469085 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.469091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.469285 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.469357 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 14:49:43.469503 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 14:49:43.469695 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25549,"mutex_wait_us":160,"thread_start_us":27641,"threads_started":1}
I20260619 14:49:43.470958 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.433483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:36757 (local address 127.0.0.1:52054)
0619 14:49:43.461741 (+ 28258us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.461752 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:43.461875 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:43.462572 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:43.462576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:43.462603 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:43.463174 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.463183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.465078 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.465085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.467127 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.467141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.469480 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.469485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.469643 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.469731 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 14:49:43.469814 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 14:49:43.470608 (+ 794us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28186,"mutex_wait_us":752,"thread_start_us":27719,"threads_started":1}
I20260619 14:49:43.471267 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.433682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.0.241.1:36757)
0619 14:49:43.460532 (+ 26850us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:43.460542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:43.461868 (+ 1326us) server_negotiation.cc:407] Connection header received
0619 14:49:43.462078 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:43.462087 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:43.462326 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:43.462541 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:43.463355 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.464916 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.467280 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.468936 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.469062 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.469257 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:43.469923 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 14:49:43.470082 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26678,"thread_start_us":26866,"threads_started":1}
I20260619 14:49:43.473745 3001 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1af98c023046b6ba1ae5157c7da93f (DEFAULT_TABLE table=test-table [id=72db1a5a535e42c3882d213662dc04c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:43.474012 2862 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1af98c023046b6ba1ae5157c7da93f (DEFAULT_TABLE table=test-table [id=72db1a5a535e42c3882d213662dc04c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:43.474686 3228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.433706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.0.241.3:42833)
0619 14:49:43.457958 (+ 24252us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:43.457971 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:43.461577 (+ 3606us) server_negotiation.cc:407] Connection header received
0619 14:49:43.461712 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:43.461720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:43.462029 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:43.462263 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:43.463327 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.464454 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.467415 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.468936 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.469065 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.469278 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:43.470224 (+ 946us) server_negotiation.cc:299] Negotiation successful
0619 14:49:43.470377 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24001,"thread_start_us":24274,"threads_started":1}
I20260619 14:49:43.475080 3001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1af98c023046b6ba1ae5157c7da93f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:43.475098 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.433090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:42833 (local address 127.0.0.1:33856)
0619 14:49:43.461481 (+ 28391us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.461496 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:43.461591 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:43.462480 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:43.462485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:43.462509 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:43.463065 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.463081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.464583 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.464591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.467232 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.467246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.469438 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.469446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.469638 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.469729 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 14:49:43.469814 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:49:43.469961 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28256,"mutex_wait_us":64,"thread_start_us":26546,"threads_started":1}
I20260619 14:49:43.475267 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1af98c023046b6ba1ae5157c7da93f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:43.478823 3144 tablet_service.cc:1467] Processing CreateTablet for tablet 4d1af98c023046b6ba1ae5157c7da93f (DEFAULT_TABLE table=test-table [id=72db1a5a535e42c3882d213662dc04c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:43.480340 3144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d1af98c023046b6ba1ae5157c7da93f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:43.485854 3229 tablet_bootstrap.cc:492] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52: Bootstrap starting.
I20260619 14:49:43.489668 3229 tablet_bootstrap.cc:654] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:43.490108 3230 tablet_bootstrap.cc:492] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65: Bootstrap starting.
I20260619 14:49:43.491071 3229 log.cc:824] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:43.493324 3230 tablet_bootstrap.cc:654] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:43.494215 3229 tablet_bootstrap.cc:492] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52: No bootstrap required, opened a new log
I20260619 14:49:43.494578 3229 ts_tablet_manager.cc:1397] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260619 14:49:43.494592 3232 tablet_bootstrap.cc:492] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9: Bootstrap starting.
I20260619 14:49:43.494858 3230 log.cc:824] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:43.497853 3230 tablet_bootstrap.cc:492] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65: No bootstrap required, opened a new log
I20260619 14:49:43.498080 3232 tablet_bootstrap.cc:654] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:43.498093 3230 ts_tablet_manager.cc:1397] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 14:49:43.499049 3232 log.cc:824] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:43.505527 3232 tablet_bootstrap.cc:492] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9: No bootstrap required, opened a new log
I20260619 14:49:43.505807 3232 ts_tablet_manager.cc:1397] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260619 14:49:43.505606 3229 raft_consensus.cc:348] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } }
I20260619 14:49:43.506060 3229 raft_consensus.cc:374] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:43.506165 3229 raft_consensus.cc:729] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c140e639feee4ea9a7daf027f7d36e52, State: Initialized, Role: FOLLOWER
I20260619 14:49:43.506795 3229 consensus_queue.cc:260] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } }
I20260619 14:49:43.508486 3229 ts_tablet_manager.cc:1428] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260619 14:49:43.508288 3230 raft_consensus.cc:348] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } }
I20260619 14:49:43.508706 3230 raft_consensus.cc:374] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:43.508810 3230 raft_consensus.cc:729] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc15d47deb04ccba2237bf49c5b8a65, State: Initialized, Role: FOLLOWER
I20260619 14:49:43.509392 3230 consensus_queue.cc:260] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } }
I20260619 14:49:43.510519 3071 heartbeater.cc:503] Master 127.0.241.62:34351 was elected leader, sending a full tablet report...
I20260619 14:49:43.511070 3230 ts_tablet_manager.cc:1428] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65: Time spent starting tablet: real 0.013s user 0.012s sys 0.003s
I20260619 14:49:43.516891 3232 raft_consensus.cc:348] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } }
I20260619 14:49:43.517186 3232 raft_consensus.cc:374] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:43.517251 3232 raft_consensus.cc:729] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2940371c6c4d84a6261123a28014d9, State: Initialized, Role: FOLLOWER
I20260619 14:49:43.517668 3232 consensus_queue.cc:260] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } }
I20260619 14:49:43.518543 3210 heartbeater.cc:503] Master 127.0.241.62:34351 was elected leader, sending a full tablet report...
I20260619 14:49:43.519117 3232 ts_tablet_manager.cc:1428] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260619 14:49:43.543222 3235 raft_consensus.cc:482] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:43.543569 3235 raft_consensus.cc:504] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } }
I20260619 14:49:43.545305 3235 leader_election.cc:288] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135), 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833)
I20260619 14:49:43.551667 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.545585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:58379 (local address 127.0.241.2:34135)
0619 14:49:43.545840 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:43.545848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:43.546303 (+ 455us) server_negotiation.cc:407] Connection header received
0619 14:49:43.546526 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:43.546534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:43.546622 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:43.546718 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:43.547370 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.548151 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.549933 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.550681 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.550835 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.550962 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:43.551242 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 14:49:43.551372 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 14:49:43.551671 3228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.545729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:55431 (local address 127.0.241.3:42833)
0619 14:49:43.545896 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:43.545901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:43.546628 (+ 727us) server_negotiation.cc:407] Connection header received
0619 14:49:43.546780 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:43.546785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:43.546861 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:43.546962 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:43.547574 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.548249 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.549988 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.550727 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.550881 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.551001 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:43.551274 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 14:49:43.551393 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:49:43.551782 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.545800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:42833 (local address 127.0.241.1:55431)
0619 14:49:43.546518 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.546534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:43.546653 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:43.546992 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:43.546995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:43.547014 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:43.547379 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.547390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.548400 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.548409 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.549831 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.549843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.550966 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.550972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.551127 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.551148 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:49:43.551251 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:49:43.551380 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":387,"threads_started":1}
I20260619 14:49:43.551882 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.545482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:34135 (local address 127.0.241.1:58379)
0619 14:49:43.546168 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.546196 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:43.546360 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:43.546760 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:43.546764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:43.546790 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:43.547208 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.547222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.548278 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.548285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.549774 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.549785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.550925 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.550932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.551090 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.551112 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:49:43.551217 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 14:49:43.551341 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":21,"thread_start_us":366,"threads_started":1}
I20260619 14:49:43.552639 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1af98c023046b6ba1ae5157c7da93f" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2940371c6c4d84a6261123a28014d9" is_pre_election: true
I20260619 14:49:43.552855 3021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1af98c023046b6ba1ae5157c7da93f" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" is_pre_election: true
I20260619 14:49:43.553149 3165 raft_consensus.cc:2454] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 0.
I20260619 14:49:43.553390 3021 raft_consensus.cc:2454] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 0.
I20260619 14:49:43.553840 2813 leader_election.cc:302] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e2940371c6c4d84a6261123a28014d9, c140e639feee4ea9a7daf027f7d36e52; no voters:
I20260619 14:49:43.554199 3235 raft_consensus.cc:2781] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:43.554391 3235 raft_consensus.cc:482] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:43.554520 3235 raft_consensus.cc:3037] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:43.556787 3235 raft_consensus.cc:504] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } }
I20260619 14:49:43.557371 3235 leader_election.cc:288] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135), 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833)
I20260619 14:49:43.557777 3021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1af98c023046b6ba1ae5157c7da93f" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc15d47deb04ccba2237bf49c5b8a65"
I20260619 14:49:43.557782 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d1af98c023046b6ba1ae5157c7da93f" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2940371c6c4d84a6261123a28014d9"
I20260619 14:49:43.557972 3165 raft_consensus.cc:3037] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:43.557973 3021 raft_consensus.cc:3037] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:43.561154 3021 raft_consensus.cc:2454] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 1.
I20260619 14:49:43.561237 3165 raft_consensus.cc:2454] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 1.
W20260619 14:49:43.561354 2933 tablet.cc:2369] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:49:43.561553 2814 leader_election.cc:302] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc15d47deb04ccba2237bf49c5b8a65, c140e639feee4ea9a7daf027f7d36e52; no voters:
I20260619 14:49:43.561961 3235 raft_consensus.cc:2781] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:43.562494 3235 raft_consensus.cc:686] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 1 LEADER]: Becoming Leader. State: Replica: c140e639feee4ea9a7daf027f7d36e52, State: Running, Role: LEADER
I20260619 14:49:43.562994 3235 consensus_queue.cc:237] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } }
I20260619 14:49:43.567629 2725 catalog_manager.cc:5495] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 reported cstate change: term changed from 0 to 1, leader changed from <none> to c140e639feee4ea9a7daf027f7d36e52 (127.0.241.1). New cstate: current_term: 1 leader_uuid: "c140e639feee4ea9a7daf027f7d36e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } health_report { overall_health: HEALTHY } } }
W20260619 14:49:43.585618 3072 tablet.cc:2369] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:49:43.601063 3211 tablet.cc:2369] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:49:43 dist-test-slave-p3ml krb5kdc[2680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:43.630695 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.617730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.0.241.62:34351)
0619 14:49:43.617920 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:43.617925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:43.618435 (+ 510us) server_negotiation.cc:407] Connection header received
0619 14:49:43.618628 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:43.618632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:43.618693 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:43.618793 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:43.619987 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.620778 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.622063 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.622462 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.622596 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.625807 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:43.625811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:43.625814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:43.625840 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:43.628105 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:43.628781 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:43.628789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:43.628793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:43.628907 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:43.629261 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:43.629268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:43.629272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:43.629532 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:43.629694 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:43.630078 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 14:49:43.630410 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":74}
I20260619 14:49:43.630728 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.617786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34351 (local address 127.0.0.1:37320)
0619 14:49:43.618302 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.618316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:43.618510 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:43.618819 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:43.618823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:43.619567 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:43.619833 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.619842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.620911 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.620919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.621913 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.621924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.622633 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.622639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.622800 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.623423 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:43.623465 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:43.625628 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:43.628235 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:43.628245 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:43.628252 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:43.628660 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:43.629013 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:43.629020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:43.629027 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:43.629160 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:43.629663 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:43.629674 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:43.629835 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 14:49:43.629996 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:49:43.630274 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":285,"threads_started":1}
I20260619 14:49:43.635718 2725 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:37320:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:49:43.636636 2725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:43.642714 3001 tablet_service.cc:1467] Processing CreateTablet for tablet 21d93abe4b614c238e567ca485b39e1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ca78ff8f58344e1ab2dd17ae0a8fa237]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:49:43.642611 2862 tablet_service.cc:1467] Processing CreateTablet for tablet 21d93abe4b614c238e567ca485b39e1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ca78ff8f58344e1ab2dd17ae0a8fa237]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:49:43.642900 3144 tablet_service.cc:1467] Processing CreateTablet for tablet 21d93abe4b614c238e567ca485b39e1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ca78ff8f58344e1ab2dd17ae0a8fa237]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:49:43.643646 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d93abe4b614c238e567ca485b39e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:43.643644 3144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d93abe4b614c238e567ca485b39e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:43.643644 3001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d93abe4b614c238e567ca485b39e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:43.649372 3229 tablet_bootstrap.cc:492] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: Bootstrap starting.
I20260619 14:49:43.650594 3230 tablet_bootstrap.cc:492] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65: Bootstrap starting.
I20260619 14:49:43.650733 3229 tablet_bootstrap.cc:654] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:43.650967 3232 tablet_bootstrap.cc:492] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9: Bootstrap starting.
I20260619 14:49:43.652237 3232 tablet_bootstrap.cc:654] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:43.652588 3230 tablet_bootstrap.cc:654] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:43.654311 3232 tablet_bootstrap.cc:492] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9: No bootstrap required, opened a new log
I20260619 14:49:43.654467 3232 ts_tablet_manager.cc:1397] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 14:49:43.654932 3229 tablet_bootstrap.cc:492] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: No bootstrap required, opened a new log
I20260619 14:49:43.655072 3229 ts_tablet_manager.cc:1397] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:49:43.655303 3232 raft_consensus.cc:348] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:43.655579 3232 raft_consensus.cc:374] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:43.655678 3232 raft_consensus.cc:729] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2940371c6c4d84a6261123a28014d9, State: Initialized, Role: FOLLOWER
I20260619 14:49:43.655742 3230 tablet_bootstrap.cc:492] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65: No bootstrap required, opened a new log
I20260619 14:49:43.655947 3230 ts_tablet_manager.cc:1397] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260619 14:49:43.655854 3229 raft_consensus.cc:348] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:43.655911 3232 consensus_queue.cc:260] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:43.656075 3229 raft_consensus.cc:374] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:43.656504 3232 ts_tablet_manager.cc:1428] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:49:43.656369 3237 tablet_replica.cc:439] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } }
I20260619 14:49:43.656172 3229 raft_consensus.cc:729] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c140e639feee4ea9a7daf027f7d36e52, State: Initialized, Role: FOLLOWER
I20260619 14:49:43.656747 3237 tablet_replica.cc:442] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:43.656648 3230 raft_consensus.cc:348] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:43.656878 3230 raft_consensus.cc:374] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:43.656975 3230 raft_consensus.cc:729] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc15d47deb04ccba2237bf49c5b8a65, State: Initialized, Role: FOLLOWER
I20260619 14:49:43.656996 3229 consensus_queue.cc:260] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:43.657440 3235 tablet_replica.cc:439] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } }
I20260619 14:49:43.657596 3229 ts_tablet_manager.cc:1428] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:49:43.657703 3235 tablet_replica.cc:442] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:43.657874 3230 consensus_queue.cc:260] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:43.658496 3230 ts_tablet_manager.cc:1428] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:49:43.658438 3236 tablet_replica.cc:439] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } }
I20260619 14:49:43.658810 3236 tablet_replica.cc:442] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:43.854712 3235 raft_consensus.cc:482] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:43.854961 3235 raft_consensus.cc:504] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:43.855777 3235 leader_election.cc:288] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135), 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833)
I20260619 14:49:43.856256 3021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d93abe4b614c238e567ca485b39e1f" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" is_pre_election: true
I20260619 14:49:43.856297 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d93abe4b614c238e567ca485b39e1f" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2940371c6c4d84a6261123a28014d9" is_pre_election: true
I20260619 14:49:43.856488 3021 raft_consensus.cc:2454] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 0.
I20260619 14:49:43.856600 3165 raft_consensus.cc:2454] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 0.
I20260619 14:49:43.856935 2814 leader_election.cc:302] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc15d47deb04ccba2237bf49c5b8a65, c140e639feee4ea9a7daf027f7d36e52; no voters:
I20260619 14:49:43.857236 3235 raft_consensus.cc:2781] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:43.857314 3235 raft_consensus.cc:482] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:43.857425 3235 raft_consensus.cc:3037] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:43.859395 3235 raft_consensus.cc:504] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:43.860025 3235 leader_election.cc:288] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135), 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833)
I20260619 14:49:43.860381 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d93abe4b614c238e567ca485b39e1f" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2940371c6c4d84a6261123a28014d9"
I20260619 14:49:43.860384 3021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d93abe4b614c238e567ca485b39e1f" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc15d47deb04ccba2237bf49c5b8a65"
I20260619 14:49:43.860589 3021 raft_consensus.cc:3037] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:43.860589 3165 raft_consensus.cc:3037] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:43.863407 3165 raft_consensus.cc:2454] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 1.
I20260619 14:49:43.863418 3021 raft_consensus.cc:2454] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 1.
I20260619 14:49:43.863858 2813 leader_election.cc:302] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e2940371c6c4d84a6261123a28014d9, c140e639feee4ea9a7daf027f7d36e52; no voters:
I20260619 14:49:43.864107 3235 raft_consensus.cc:2781] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:43.864267 3235 raft_consensus.cc:686] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [term 1 LEADER]: Becoming Leader. State: Replica: c140e639feee4ea9a7daf027f7d36e52, State: Running, Role: LEADER
I20260619 14:49:43.864502 3235 consensus_queue.cc:237] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:43.866309 3240 tablet_replica.cc:439] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: TxnStatusTablet state changed. Reason: New leader c140e639feee4ea9a7daf027f7d36e52. Latest consensus state: current_term: 1 leader_uuid: "c140e639feee4ea9a7daf027f7d36e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } }
I20260619 14:49:43.866762 3240 tablet_replica.cc:442] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:49:43.867385 3252 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:49:43.867727 3252 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:49:43.868221 2723 catalog_manager.cc:5495] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 reported cstate change: term changed from 0 to 1, leader changed from <none> to c140e639feee4ea9a7daf027f7d36e52 (127.0.241.1). New cstate: current_term: 1 leader_uuid: "c140e639feee4ea9a7daf027f7d36e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } health_report { overall_health: UNKNOWN } } }
I20260619 14:49:43.894688 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.888704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.0.241.1:36757)
0619 14:49:43.888872 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:43.888879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:43.888966 (+ 87us) server_negotiation.cc:407] Connection header received
0619 14:49:43.889218 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:43.889222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:43.889310 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:43.889427 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:43.889827 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.890583 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.892481 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.892886 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.892977 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.893501 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:43.894149 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:43.894313 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:43.894410 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 14:49:43.894470 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:49:43.894842 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.888615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:36757 (local address 127.0.0.1:52070)
0619 14:49:43.888869 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.888890 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:43.889039 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:43.889481 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:43.889486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:43.889499 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:49:43.889697 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.889708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.890708 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.890715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.892353 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.892368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.893040 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.893046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.893243 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.893314 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:49:43.894273 (+ 959us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:49:43.894294 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:49:43.894385 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 14:49:43.894536 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 14:49:43.902263 3165 raft_consensus.cc:1261] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Refusing update from remote peer c140e639feee4ea9a7daf027f7d36e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:43.902262 3021 raft_consensus.cc:1261] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Refusing update from remote peer c140e639feee4ea9a7daf027f7d36e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:43.903223 3240 consensus_queue.cc:1035] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:43.903581 3235 consensus_queue.cc:1035] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:43.914273 3237 tablet_replica.cc:439] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9: TxnStatusTablet state changed. Reason: New leader c140e639feee4ea9a7daf027f7d36e52. Latest consensus state: current_term: 1 leader_uuid: "c140e639feee4ea9a7daf027f7d36e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } }
I20260619 14:49:43.914559 3237 tablet_replica.cc:442] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:43.915184 3236 tablet_replica.cc:439] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65: TxnStatusTablet state changed. Reason: New leader c140e639feee4ea9a7daf027f7d36e52. Latest consensus state: current_term: 1 leader_uuid: "c140e639feee4ea9a7daf027f7d36e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } }
I20260619 14:49:43.915462 3236 tablet_replica.cc:442] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:43.919168 3235 tablet_replica.cc:439] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c140e639feee4ea9a7daf027f7d36e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } }
I20260619 14:49:43.919483 3235 tablet_replica.cc:442] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:49:43.919713 3235 tablet_replica.cc:439] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c140e639feee4ea9a7daf027f7d36e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } }
I20260619 14:49:43.919967 3235 tablet_replica.cc:442] T 21d93abe4b614c238e567ca485b39e1f P c140e639feee4ea9a7daf027f7d36e52: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:49:43.920174 3257 mvcc.cc:205] Tried to move back new op lower bound from 7298582871653908480 to 7298582871511552000. Current Snapshot: MvccSnapshot[applied={T|T < 7298582871653908480}]
I20260619 14:49:43.926304 3236 tablet_replica.cc:439] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c140e639feee4ea9a7daf027f7d36e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } }
I20260619 14:49:43.926555 3236 tablet_replica.cc:442] T 21d93abe4b614c238e567ca485b39e1f P 6cc15d47deb04ccba2237bf49c5b8a65: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:43.932897 3237 tablet_replica.cc:439] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c140e639feee4ea9a7daf027f7d36e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } } }
I20260619 14:49:43.933138 3237 tablet_replica.cc:442] T 21d93abe4b614c238e567ca485b39e1f P 4e2940371c6c4d84a6261123a28014d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:49:43.942238 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.936476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.0.241.1:36757)
0619 14:49:43.936651 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:43.936657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:43.937003 (+ 346us) server_negotiation.cc:407] Connection header received
0619 14:49:43.937144 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:43.937149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:43.937229 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:43.937328 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:43.937698 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.938853 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.940358 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.941038 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.941193 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.941521 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:43.941663 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:43.941756 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:43.941858 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 14:49:43.941932 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:49:43.959298 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.952489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:55131 (local address 127.0.241.1:36757)
0619 14:49:43.952654 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:43.952660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:43.953447 (+ 787us) server_negotiation.cc:407] Connection header received
0619 14:49:43.953617 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:43.953623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:43.953694 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:43.953776 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:43.954443 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.955414 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.957661 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.958080 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.958169 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.958697 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:43.958778 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:43.958836 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:43.958990 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 14:49:43.959046 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:49:43.959635 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.952496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:36757 (local address 127.0.241.1:55131)
0619 14:49:43.953332 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.953357 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:43.953492 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:43.953910 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:43.953914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:43.953925 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:49:43.954267 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.954276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.955607 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.955612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.957520 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:43.957534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:43.958249 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:43.958255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:43.958443 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:43.958514 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:49:43.958873 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:49:43.958884 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 14:49:43.958966 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:49:43.959110 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":431,"threads_started":1}
I20260619 14:49:43.968439 3165 raft_consensus.cc:1261] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Refusing update from remote peer c140e639feee4ea9a7daf027f7d36e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:43.968502 3021 raft_consensus.cc:1261] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Refusing update from remote peer c140e639feee4ea9a7daf027f7d36e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:43.969017 3240 consensus_queue.cc:1035] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:43.969345 3261 consensus_queue.cc:1035] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:43.975229 3255 mvcc.cc:205] Tried to move back new op lower bound from 7298582871930757120 to 7298582870277738496. Current Snapshot: MvccSnapshot[applied={T|T < 7298582871930757120}]
I20260619 14:49:43.978184 3256 mvcc.cc:205] Tried to move back new op lower bound from 7298582871930757120 to 7298582870277738496. Current Snapshot: MvccSnapshot[applied={T|T < 7298582871930757120}]
I20260619 14:49:44.029155 2723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:44.029438 2723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:44.032830 2723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e: Sending DeleteTablet for 3 replicas of tablet 4d1af98c023046b6ba1ae5157c7da93f
I20260619 14:49:44.033787 3001 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1af98c023046b6ba1ae5157c7da93f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:44 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:36292
I20260619 14:49:44.034181 3144 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1af98c023046b6ba1ae5157c7da93f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:44 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:33856
I20260619 14:49:44.034437 2862 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d1af98c023046b6ba1ae5157c7da93f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:44 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:52054
I20260619 14:49:44.036371 2723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:49:44.037135 2723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:44.045043 3139 tablet_service.cc:1467] Processing CreateTablet for tablet d1c6ea4eafac430ba3aaa585df63a249 (DEFAULT_TABLE table=test-table [id=3bc526da641e4e5c804a91d8b47f97ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:44.045089 2861 tablet_service.cc:1467] Processing CreateTablet for tablet d1c6ea4eafac430ba3aaa585df63a249 (DEFAULT_TABLE table=test-table [id=3bc526da641e4e5c804a91d8b47f97ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:44.045562 3139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c6ea4eafac430ba3aaa585df63a249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:44.045459 3000 tablet_service.cc:1467] Processing CreateTablet for tablet d1c6ea4eafac430ba3aaa585df63a249 (DEFAULT_TABLE table=test-table [id=3bc526da641e4e5c804a91d8b47f97ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:44.045795 2861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c6ea4eafac430ba3aaa585df63a249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:44.045928 3000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c6ea4eafac430ba3aaa585df63a249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:44.049026 3274 tablet_replica.cc:330] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52: stopping tablet replica
I20260619 14:49:44.049433 3274 raft_consensus.cc:2229] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:49:44.050045 3274 raft_consensus.cc:2258] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:44.050817 3273 tablet_replica.cc:330] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9: stopping tablet replica
I20260619 14:49:44.051244 3273 raft_consensus.cc:2229] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:44.051613 3273 raft_consensus.cc:2258] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:44.052086 3232 tablet_bootstrap.cc:492] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9: Bootstrap starting.
I20260619 14:49:44.053100 3230 tablet_bootstrap.cc:492] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65: Bootstrap starting.
I20260619 14:49:44.053795 3232 tablet_bootstrap.cc:654] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:44.054801 3230 tablet_bootstrap.cc:654] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:44.055558 3273 ts_tablet_manager.cc:1905] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:44.055685 3229 tablet_bootstrap.cc:492] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52: Bootstrap starting.
I20260619 14:49:44.056494 3274 ts_tablet_manager.cc:1905] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:44.057627 3229 tablet_bootstrap.cc:654] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:44.064039 3273 ts_tablet_manager.cc:1918] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:49:44.064248 3273 log.cc:1196] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/4d1af98c023046b6ba1ae5157c7da93f
I20260619 14:49:44.064857 3273 ts_tablet_manager.cc:1939] T 4d1af98c023046b6ba1ae5157c7da93f P 4e2940371c6c4d84a6261123a28014d9: Deleting consensus metadata
I20260619 14:49:44.066823 2710 catalog_manager.cc:4841] TS 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833): tablet 4d1af98c023046b6ba1ae5157c7da93f (table test-table [id=72db1a5a535e42c3882d213662dc04c4]) successfully deleted
I20260619 14:49:44.071653 3274 ts_tablet_manager.cc:1918] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:49:44.071830 3274 log.cc:1196] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/4d1af98c023046b6ba1ae5157c7da93f
I20260619 14:49:44.072439 3274 ts_tablet_manager.cc:1939] T 4d1af98c023046b6ba1ae5157c7da93f P c140e639feee4ea9a7daf027f7d36e52: Deleting consensus metadata
I20260619 14:49:44.072685 3230 tablet_bootstrap.cc:492] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65: No bootstrap required, opened a new log
I20260619 14:49:44.072852 3230 ts_tablet_manager.cc:1397] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.004s
I20260619 14:49:44.073601 3230 raft_consensus.cc:348] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:44.073854 3230 raft_consensus.cc:374] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:44.073935 3230 raft_consensus.cc:729] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc15d47deb04ccba2237bf49c5b8a65, State: Initialized, Role: FOLLOWER
I20260619 14:49:44.074218 3230 consensus_queue.cc:260] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:44.074927 2712 catalog_manager.cc:4841] TS c140e639feee4ea9a7daf027f7d36e52 (127.0.241.1:36757): tablet 4d1af98c023046b6ba1ae5157c7da93f (table test-table [id=72db1a5a535e42c3882d213662dc04c4]) successfully deleted
I20260619 14:49:44.075703 3230 ts_tablet_manager.cc:1428] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 14:49:44.077917 3229 tablet_bootstrap.cc:492] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52: No bootstrap required, opened a new log
I20260619 14:49:44.078141 3229 ts_tablet_manager.cc:1397] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.010s
I20260619 14:49:44.078982 3272 tablet_replica.cc:330] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65: stopping tablet replica
I20260619 14:49:44.078984 3229 raft_consensus.cc:348] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:44.079365 3229 raft_consensus.cc:374] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:44.079509 3229 raft_consensus.cc:729] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c140e639feee4ea9a7daf027f7d36e52, State: Initialized, Role: FOLLOWER
I20260619 14:49:44.079818 3229 consensus_queue.cc:260] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:44.080456 3232 tablet_bootstrap.cc:492] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9: No bootstrap required, opened a new log
I20260619 14:49:44.080552 3229 ts_tablet_manager.cc:1428] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 14:49:44.080628 3232 ts_tablet_manager.cc:1397] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.005s
I20260619 14:49:44.079424 3272 raft_consensus.cc:2229] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:44.081334 3272 raft_consensus.cc:2258] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:44.081405 3232 raft_consensus.cc:348] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:44.081641 3232 raft_consensus.cc:374] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:44.081720 3232 raft_consensus.cc:729] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2940371c6c4d84a6261123a28014d9, State: Initialized, Role: FOLLOWER
I20260619 14:49:44.081941 3232 consensus_queue.cc:260] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:44.082442 3232 ts_tablet_manager.cc:1428] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 14:49:44.084188 3272 ts_tablet_manager.cc:1905] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:44.089511 3272 ts_tablet_manager.cc:1918] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:49:44.089622 3272 log.cc:1196] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/4d1af98c023046b6ba1ae5157c7da93f
I20260619 14:49:44.090050 3272 ts_tablet_manager.cc:1939] T 4d1af98c023046b6ba1ae5157c7da93f P 6cc15d47deb04ccba2237bf49c5b8a65: Deleting consensus metadata
I20260619 14:49:44.091383 2711 catalog_manager.cc:4841] TS 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135): tablet 4d1af98c023046b6ba1ae5157c7da93f (table test-table [id=72db1a5a535e42c3882d213662dc04c4]) successfully deleted
I20260619 14:49:44.358170 3236 raft_consensus.cc:482] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:44.358378 3236 raft_consensus.cc:504] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:44.359880 3236 leader_election.cc:288] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c140e639feee4ea9a7daf027f7d36e52 (127.0.241.1:36757), 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833)
I20260619 14:49:44.362638 3240 raft_consensus.cc:482] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:44.362807 3240 raft_consensus.cc:504] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:44.363515 3240 leader_election.cc:288] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135), 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833)
I20260619 14:49:44.364284 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c6ea4eafac430ba3aaa585df63a249" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2940371c6c4d84a6261123a28014d9" is_pre_election: true
I20260619 14:49:44.364516 3165 raft_consensus.cc:2454] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 0.
I20260619 14:49:44.364955 2813 leader_election.cc:302] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e2940371c6c4d84a6261123a28014d9, c140e639feee4ea9a7daf027f7d36e52; no voters:
I20260619 14:49:44.365212 3240 raft_consensus.cc:2781] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:44.365321 3240 raft_consensus.cc:482] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:44.365446 3240 raft_consensus.cc:3037] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:44.368533 3240 raft_consensus.cc:504] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:44.369704 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c6ea4eafac430ba3aaa585df63a249" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2940371c6c4d84a6261123a28014d9"
I20260619 14:49:44.369908 3165 raft_consensus.cc:3037] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:44.372869 3165 raft_consensus.cc:2454] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 1.
I20260619 14:49:44.373282 2813 leader_election.cc:302] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e2940371c6c4d84a6261123a28014d9, c140e639feee4ea9a7daf027f7d36e52; no voters:
I20260619 14:49:44.373559 3261 raft_consensus.cc:2781] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:44.373728 3261 raft_consensus.cc:686] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 1 LEADER]: Becoming Leader. State: Replica: c140e639feee4ea9a7daf027f7d36e52, State: Running, Role: LEADER
I20260619 14:49:44.373827 3020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c6ea4eafac430ba3aaa585df63a249" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" is_pre_election: true
I20260619 14:49:44.374106 3020 raft_consensus.cc:2454] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 0.
I20260619 14:49:44.373984 3261 consensus_queue.cc:237] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:44.374722 3020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c6ea4eafac430ba3aaa585df63a249" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc15d47deb04ccba2237bf49c5b8a65"
I20260619 14:49:44.374938 3020 raft_consensus.cc:3037] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:44.375737 3240 leader_election.cc:288] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135), 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833)
I20260619 14:49:44.378619 3020 raft_consensus.cc:2454] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 1.
I20260619 14:49:44.386155 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:44.360486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:42833 (local address 127.0.241.2:39517)
0619 14:49:44.378211 (+ 17725us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:44.378230 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:44.378353 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:44.379190 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:44.379196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:44.379229 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:44.379588 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:44.379601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:44.381247 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:44.381252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:44.383520 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:44.383539 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:44.385288 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:44.385294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:44.385470 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:44.385504 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:49:44.385587 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 14:49:44.385723 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17610,"mutex_wait_us":1,"thread_start_us":18507,"threads_started":1}
I20260619 14:49:44.386890 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:44.360328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:32841 (local address 127.0.241.1:36757)
0619 14:49:44.360573 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:44.360580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:44.373225 (+ 12645us) server_negotiation.cc:407] Connection header received
0619 14:49:44.379701 (+ 6476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:44.379709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:44.379786 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:44.379850 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:44.380621 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:44.381730 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:44.384225 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:44.385665 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:44.385775 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:44.385875 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:44.386467 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 14:49:44.386602 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 14:49:44.386916 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:44.360999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:39517 (local address 127.0.241.3:42833)
0619 14:49:44.370273 (+ 9274us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:44.370282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:44.378473 (+ 8191us) server_negotiation.cc:407] Connection header received
0619 14:49:44.378541 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:44.378546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:44.378642 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:44.378804 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:44.379761 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:44.381057 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:44.383688 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:44.384928 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:44.385027 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:44.385158 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:44.386446 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0619 14:49:44.386587 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9078,"thread_start_us":8920,"threads_started":1}
I20260619 14:49:44.387630 3278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:44.360040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:36757 (local address 127.0.241.2:32841)
0619 14:49:44.373096 (+ 13056us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:44.373123 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:44.379520 (+ 6397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:44.380044 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:44.380048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:44.380076 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:44.380407 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:44.380417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:44.381933 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:44.381938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:44.384041 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:44.384054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:44.386061 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:44.386066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:44.386274 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:44.386310 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:49:44.387124 (+ 814us) client_negotiation.cc:240] Negotiation successful
0619 14:49:44.387270 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12908,"thread_start_us":13218,"threads_started":1}
I20260619 14:49:44.387703 2724 catalog_manager.cc:5495] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 reported cstate change: term changed from 0 to 1, leader changed from <none> to c140e639feee4ea9a7daf027f7d36e52 (127.0.241.1). New cstate: current_term: 1 leader_uuid: "c140e639feee4ea9a7daf027f7d36e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } health_report { overall_health: UNKNOWN } } }
I20260619 14:49:44.388423 2887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c6ea4eafac430ba3aaa585df63a249" candidate_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c140e639feee4ea9a7daf027f7d36e52" is_pre_election: true
I20260619 14:49:44.388893 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c6ea4eafac430ba3aaa585df63a249" candidate_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2940371c6c4d84a6261123a28014d9" is_pre_election: true
I20260619 14:49:44.389240 3165 raft_consensus.cc:2379] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cc15d47deb04ccba2237bf49c5b8a65 in current term 1: Already voted for candidate c140e639feee4ea9a7daf027f7d36e52 in this term.
I20260619 14:49:44.389734 2953 leader_election.cc:302] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc15d47deb04ccba2237bf49c5b8a65; no voters: 4e2940371c6c4d84a6261123a28014d9, c140e639feee4ea9a7daf027f7d36e52
I20260619 14:49:44.390377 3236 raft_consensus.cc:2726] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 14:49:44.440187 3164 raft_consensus.cc:1261] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Refusing update from remote peer c140e639feee4ea9a7daf027f7d36e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:44.440703 3261 consensus_queue.cc:1035] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:44.440563 3020 raft_consensus.cc:1261] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Refusing update from remote peer c140e639feee4ea9a7daf027f7d36e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:44.441569 3261 consensus_queue.cc:1035] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:49:44.465817 3255 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:49:46.043681 2724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:46.043926 2724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:46.048267 2724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e: Sending DeleteTablet for 3 replicas of tablet d1c6ea4eafac430ba3aaa585df63a249
I20260619 14:49:46.049029 2861 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c6ea4eafac430ba3aaa585df63a249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:46 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:52054
I20260619 14:49:46.049136 3001 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c6ea4eafac430ba3aaa585df63a249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:46 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:36292
I20260619 14:49:46.049285 3139 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c6ea4eafac430ba3aaa585df63a249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:46 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:33856
I20260619 14:49:46.061002 3306 tablet_replica.cc:330] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9: stopping tablet replica
I20260619 14:49:46.061199 3305 tablet_replica.cc:330] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65: stopping tablet replica
I20260619 14:49:46.061452 3305 raft_consensus.cc:2229] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:46.061738 3305 raft_consensus.cc:2258] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:46.061247 3306 raft_consensus.cc:2229] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:46.062299 3304 tablet_replica.cc:330] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52: stopping tablet replica
I20260619 14:49:46.062322 3306 raft_consensus.cc:2258] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:46.062513 3304 raft_consensus.cc:2229] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:49:46.062876 3304 raft_consensus.cc:2258] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:46.063146 3305 ts_tablet_manager.cc:1905] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:46.063277 3306 ts_tablet_manager.cc:1905] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:46.063823 3304 ts_tablet_manager.cc:1905] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 14:49:46 dist-test-slave-p3ml krb5kdc[2680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260619 14:49:46.157627 964 mini_kdc.cc:317] Time spent kinit for test-user: real 0.108s user 0.001s sys 0.011s
I20260619 14:49:46.161192 3305 ts_tablet_manager.cc:1918] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:49:46.161192 3304 ts_tablet_manager.cc:1918] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:49:46.161355 3305 log.cc:1196] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/d1c6ea4eafac430ba3aaa585df63a249
I20260619 14:49:46.161345 3304 log.cc:1196] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/d1c6ea4eafac430ba3aaa585df63a249
I20260619 14:49:46.161787 3305 ts_tablet_manager.cc:1939] T d1c6ea4eafac430ba3aaa585df63a249 P 6cc15d47deb04ccba2237bf49c5b8a65: Deleting consensus metadata
I20260619 14:49:46.161787 3304 ts_tablet_manager.cc:1939] T d1c6ea4eafac430ba3aaa585df63a249 P c140e639feee4ea9a7daf027f7d36e52: Deleting consensus metadata
I20260619 14:49:46.164264 3306 ts_tablet_manager.cc:1918] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:49:46.164443 3306 log.cc:1196] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/d1c6ea4eafac430ba3aaa585df63a249
I20260619 14:49:46.164850 3306 ts_tablet_manager.cc:1939] T d1c6ea4eafac430ba3aaa585df63a249 P 4e2940371c6c4d84a6261123a28014d9: Deleting consensus metadata
Jun 19 14:49:46 dist-test-slave-p3ml krb5kdc[2680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:46.183568 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:46.165115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.0.241.62:34351)
0619 14:49:46.166130 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:46.166134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:46.166155 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:49:46.166211 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:46.166216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:46.166280 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:46.166374 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:46.167786 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.168562 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.169802 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.170179 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.170270 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:46.178531 (+ 8261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:46.178555 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:46.178558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:46.178591 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:46.181219 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:46.181825 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:46.181829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:46.181832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:46.181953 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:46.182299 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:46.182303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:46.182306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:46.182602 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:46.182784 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:46.183047 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 14:49:46.183201 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":700,"threads_started":1}
I20260619 14:49:46.186846 2724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:49:46.187495 2724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:46.193606 2861 tablet_service.cc:1467] Processing CreateTablet for tablet 1319f9abf0ca4b14a650c4ddad7b3e86 (DEFAULT_TABLE table=test-table [id=75da6945f98b4b50b30af92b318e2fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:46.193819 3139 tablet_service.cc:1467] Processing CreateTablet for tablet 1319f9abf0ca4b14a650c4ddad7b3e86 (DEFAULT_TABLE table=test-table [id=75da6945f98b4b50b30af92b318e2fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:46.194080 2861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1319f9abf0ca4b14a650c4ddad7b3e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:46.194305 3139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1319f9abf0ca4b14a650c4ddad7b3e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:46.194442 3001 tablet_service.cc:1467] Processing CreateTablet for tablet 1319f9abf0ca4b14a650c4ddad7b3e86 (DEFAULT_TABLE table=test-table [id=75da6945f98b4b50b30af92b318e2fc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:46.194851 3001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1319f9abf0ca4b14a650c4ddad7b3e86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:46.199587 3315 tablet_bootstrap.cc:492] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9: Bootstrap starting.
I20260619 14:49:46.201485 3315 tablet_bootstrap.cc:654] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:46.202147 3317 tablet_bootstrap.cc:492] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65: Bootstrap starting.
I20260619 14:49:46.202929 3316 tablet_bootstrap.cc:492] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52: Bootstrap starting.
I20260619 14:49:46.204778 3317 tablet_bootstrap.cc:654] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:46.204983 3315 tablet_bootstrap.cc:492] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9: No bootstrap required, opened a new log
I20260619 14:49:46.204993 3316 tablet_bootstrap.cc:654] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:46.205134 3315 ts_tablet_manager.cc:1397] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 14:49:46.205940 3315 raft_consensus.cc:348] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.206172 3315 raft_consensus.cc:374] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:46.206264 3315 raft_consensus.cc:729] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2940371c6c4d84a6261123a28014d9, State: Initialized, Role: FOLLOWER
I20260619 14:49:46.206557 3315 consensus_queue.cc:260] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.206981 3317 tablet_bootstrap.cc:492] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65: No bootstrap required, opened a new log
I20260619 14:49:46.207137 3317 ts_tablet_manager.cc:1397] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:49:46.207500 3315 ts_tablet_manager.cc:1428] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 14:49:46.207913 3317 raft_consensus.cc:348] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.208154 3317 raft_consensus.cc:374] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:46.208899 3317 raft_consensus.cc:729] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc15d47deb04ccba2237bf49c5b8a65, State: Initialized, Role: FOLLOWER
I20260619 14:49:46.209151 3317 consensus_queue.cc:260] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.210005 3316 tablet_bootstrap.cc:492] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52: No bootstrap required, opened a new log
I20260619 14:49:46.210000 3317 ts_tablet_manager.cc:1428] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:46.210155 3316 ts_tablet_manager.cc:1397] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260619 14:49:46.211146 3316 raft_consensus.cc:348] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.211416 3316 raft_consensus.cc:374] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:46.211540 3316 raft_consensus.cc:729] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c140e639feee4ea9a7daf027f7d36e52, State: Initialized, Role: FOLLOWER
I20260619 14:49:46.211863 3316 consensus_queue.cc:260] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.212975 3316 ts_tablet_manager.cc:1428] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 14:49:46.215893 2710 catalog_manager.cc:4841] TS 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833): tablet d1c6ea4eafac430ba3aaa585df63a249 (table test-table [id=3bc526da641e4e5c804a91d8b47f97ae]) successfully deleted
I20260619 14:49:46.216039 2711 catalog_manager.cc:4841] TS 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135): tablet d1c6ea4eafac430ba3aaa585df63a249 (table test-table [id=3bc526da641e4e5c804a91d8b47f97ae]) successfully deleted
I20260619 14:49:46.216198 2712 catalog_manager.cc:4841] TS c140e639feee4ea9a7daf027f7d36e52 (127.0.241.1:36757): tablet d1c6ea4eafac430ba3aaa585df63a249 (table test-table [id=3bc526da641e4e5c804a91d8b47f97ae]) successfully deleted
I20260619 14:49:46.345650 3240 raft_consensus.cc:482] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:46.345846 3240 raft_consensus.cc:504] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.346555 3240 leader_election.cc:288] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135), 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833)
I20260619 14:49:46.347091 3020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1319f9abf0ca4b14a650c4ddad7b3e86" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" is_pre_election: true
I20260619 14:49:46.347101 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1319f9abf0ca4b14a650c4ddad7b3e86" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2940371c6c4d84a6261123a28014d9" is_pre_election: true
I20260619 14:49:46.347332 3020 raft_consensus.cc:2454] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 0.
I20260619 14:49:46.347361 3165 raft_consensus.cc:2454] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 0.
I20260619 14:49:46.347834 2814 leader_election.cc:302] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc15d47deb04ccba2237bf49c5b8a65, c140e639feee4ea9a7daf027f7d36e52; no voters:
I20260619 14:49:46.348094 3240 raft_consensus.cc:2781] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:46.348209 3240 raft_consensus.cc:482] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:46.348294 3240 raft_consensus.cc:3037] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:46.350829 3240 raft_consensus.cc:504] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.351466 3240 leader_election.cc:288] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135), 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833)
I20260619 14:49:46.351882 3020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1319f9abf0ca4b14a650c4ddad7b3e86" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc15d47deb04ccba2237bf49c5b8a65"
I20260619 14:49:46.351950 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1319f9abf0ca4b14a650c4ddad7b3e86" candidate_uuid: "c140e639feee4ea9a7daf027f7d36e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2940371c6c4d84a6261123a28014d9"
I20260619 14:49:46.352075 3020 raft_consensus.cc:3037] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:46.352185 3165 raft_consensus.cc:3037] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:46.354959 3020 raft_consensus.cc:2454] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 1.
I20260619 14:49:46.355155 3165 raft_consensus.cc:2454] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c140e639feee4ea9a7daf027f7d36e52 in term 1.
I20260619 14:49:46.355348 2814 leader_election.cc:302] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc15d47deb04ccba2237bf49c5b8a65, c140e639feee4ea9a7daf027f7d36e52; no voters:
I20260619 14:49:46.355752 3240 raft_consensus.cc:2781] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:46.355921 3240 raft_consensus.cc:686] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 1 LEADER]: Becoming Leader. State: Replica: c140e639feee4ea9a7daf027f7d36e52, State: Running, Role: LEADER
I20260619 14:49:46.356185 3240 consensus_queue.cc:237] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.358917 2724 catalog_manager.cc:5495] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 reported cstate change: term changed from 0 to 1, leader changed from <none> to c140e639feee4ea9a7daf027f7d36e52 (127.0.241.1). New cstate: current_term: 1 leader_uuid: "c140e639feee4ea9a7daf027f7d36e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } health_report { overall_health: UNKNOWN } } }
I20260619 14:49:46.402801 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:46.397713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52088 (local address 127.0.241.1:36757)
0619 14:49:46.398428 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:46.398432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:46.398449 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:49:46.398494 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:46.398497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:46.398549 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:46.398624 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:46.398990 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.399811 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.401088 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.401493 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.401580 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:46.402106 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:46.402201 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:46.402264 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:46.402400 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 14:49:46.402453 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":508,"threads_started":1}
I20260619 14:49:46.411041 3020 raft_consensus.cc:1261] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Refusing update from remote peer c140e639feee4ea9a7daf027f7d36e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:46.411054 3165 raft_consensus.cc:1261] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Refusing update from remote peer c140e639feee4ea9a7daf027f7d36e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:46.411655 3287 consensus_queue.cc:1035] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:46.411934 3293 consensus_queue.cc:1035] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:46.451915 2723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:46.452107 2723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:46.454735 2723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e: Sending DeleteTablet for 3 replicas of tablet 1319f9abf0ca4b14a650c4ddad7b3e86
I20260619 14:49:46.455384 3001 tablet_service.cc:1514] Processing DeleteTablet for tablet 1319f9abf0ca4b14a650c4ddad7b3e86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:46 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:36292
I20260619 14:49:46.455541 3139 tablet_service.cc:1514] Processing DeleteTablet for tablet 1319f9abf0ca4b14a650c4ddad7b3e86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:46 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:33856
I20260619 14:49:46.455569 2862 tablet_service.cc:1514] Processing DeleteTablet for tablet 1319f9abf0ca4b14a650c4ddad7b3e86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:46 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:52054
I20260619 14:49:46.455672 3305 tablet_replica.cc:330] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65: stopping tablet replica
I20260619 14:49:46.455791 3304 tablet_replica.cc:330] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52: stopping tablet replica
I20260619 14:49:46.455873 3305 raft_consensus.cc:2229] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:46.455971 3304 raft_consensus.cc:2229] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:49:46.456058 3305 raft_consensus.cc:2258] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:46.456243 3306 tablet_replica.cc:330] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9: stopping tablet replica
I20260619 14:49:46.456231 3304 raft_consensus.cc:2258] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:46.456490 3306 raft_consensus.cc:2229] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:46.456683 3306 raft_consensus.cc:2258] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:46.457165 3305 ts_tablet_manager.cc:1905] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:46.457255 3304 ts_tablet_manager.cc:1905] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:46.457563 3306 ts_tablet_manager.cc:1905] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:46.462222 3306 ts_tablet_manager.cc:1918] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:49:46.462376 3306 log.cc:1196] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/1319f9abf0ca4b14a650c4ddad7b3e86
I20260619 14:49:46.462721 3306 ts_tablet_manager.cc:1939] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 4e2940371c6c4d84a6261123a28014d9: Deleting consensus metadata
I20260619 14:49:46.463361 3305 ts_tablet_manager.cc:1918] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:49:46.463361 3304 ts_tablet_manager.cc:1918] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:49:46.463514 3305 log.cc:1196] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/1319f9abf0ca4b14a650c4ddad7b3e86
I20260619 14:49:46.463513 3304 log.cc:1196] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/1319f9abf0ca4b14a650c4ddad7b3e86
I20260619 14:49:46.463505 2710 catalog_manager.cc:4841] TS 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833): tablet 1319f9abf0ca4b14a650c4ddad7b3e86 (table test-table [id=75da6945f98b4b50b30af92b318e2fc1]) successfully deleted
I20260619 14:49:46.463923 3305 ts_tablet_manager.cc:1939] T 1319f9abf0ca4b14a650c4ddad7b3e86 P 6cc15d47deb04ccba2237bf49c5b8a65: Deleting consensus metadata
I20260619 14:49:46.463943 3304 ts_tablet_manager.cc:1939] T 1319f9abf0ca4b14a650c4ddad7b3e86 P c140e639feee4ea9a7daf027f7d36e52: Deleting consensus metadata
I20260619 14:49:46.464818 2711 catalog_manager.cc:4841] TS 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135): tablet 1319f9abf0ca4b14a650c4ddad7b3e86 (table test-table [id=75da6945f98b4b50b30af92b318e2fc1]) successfully deleted
I20260619 14:49:46.464946 2712 catalog_manager.cc:4841] TS c140e639feee4ea9a7daf027f7d36e52 (127.0.241.1:36757): tablet 1319f9abf0ca4b14a650c4ddad7b3e86 (table test-table [id=75da6945f98b4b50b30af92b318e2fc1]) successfully deleted
I20260619 14:49:46.465966 2723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:49:46.466575 2723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:46.472659 3001 tablet_service.cc:1467] Processing CreateTablet for tablet 97c92a3764b8418e85d6d44e21a1ea77 (DEFAULT_TABLE table=test-table [id=ade680d27e3a4395a81eaffbe448bf6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:46.472690 3139 tablet_service.cc:1467] Processing CreateTablet for tablet 97c92a3764b8418e85d6d44e21a1ea77 (DEFAULT_TABLE table=test-table [id=ade680d27e3a4395a81eaffbe448bf6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:46.472867 2862 tablet_service.cc:1467] Processing CreateTablet for tablet 97c92a3764b8418e85d6d44e21a1ea77 (DEFAULT_TABLE table=test-table [id=ade680d27e3a4395a81eaffbe448bf6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:46.473079 3139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c92a3764b8418e85d6d44e21a1ea77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:46.473083 3001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c92a3764b8418e85d6d44e21a1ea77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:46.473233 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c92a3764b8418e85d6d44e21a1ea77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:46.477408 3317 tablet_bootstrap.cc:492] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65: Bootstrap starting.
I20260619 14:49:46.478704 3317 tablet_bootstrap.cc:654] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:46.479599 3316 tablet_bootstrap.cc:492] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52: Bootstrap starting.
I20260619 14:49:46.479676 3315 tablet_bootstrap.cc:492] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9: Bootstrap starting.
I20260619 14:49:46.480731 3316 tablet_bootstrap.cc:654] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:46.480831 3315 tablet_bootstrap.cc:654] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:46.483057 3317 tablet_bootstrap.cc:492] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65: No bootstrap required, opened a new log
I20260619 14:49:46.483091 3315 tablet_bootstrap.cc:492] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9: No bootstrap required, opened a new log
I20260619 14:49:46.483239 3317 ts_tablet_manager.cc:1397] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 14:49:46.483232 3315 ts_tablet_manager.cc:1397] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 14:49:46.483419 3316 tablet_bootstrap.cc:492] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52: No bootstrap required, opened a new log
I20260619 14:49:46.483575 3316 ts_tablet_manager.cc:1397] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260619 14:49:46.484371 3315 raft_consensus.cc:348] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.484354 3316 raft_consensus.cc:348] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.484609 3316 raft_consensus.cc:374] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:46.484609 3315 raft_consensus.cc:374] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:46.484688 3316 raft_consensus.cc:729] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c140e639feee4ea9a7daf027f7d36e52, State: Initialized, Role: FOLLOWER
I20260619 14:49:46.484689 3315 raft_consensus.cc:729] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2940371c6c4d84a6261123a28014d9, State: Initialized, Role: FOLLOWER
I20260619 14:49:46.484923 3315 consensus_queue.cc:260] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.484922 3316 consensus_queue.cc:260] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.484282 3317 raft_consensus.cc:348] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.485422 3317 raft_consensus.cc:374] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:46.485459 3316 ts_tablet_manager.cc:1428] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 14:49:46.485532 3317 raft_consensus.cc:729] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc15d47deb04ccba2237bf49c5b8a65, State: Initialized, Role: FOLLOWER
I20260619 14:49:46.485764 3317 consensus_queue.cc:260] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.486641 3315 ts_tablet_manager.cc:1428] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 14:49:46.487524 3317 ts_tablet_manager.cc:1428] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260619 14:49:46.674510 3322 raft_consensus.cc:482] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:46.674739 3322 raft_consensus.cc:504] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.675508 3322 leader_election.cc:288] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c140e639feee4ea9a7daf027f7d36e52 (127.0.241.1:36757), 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833)
I20260619 14:49:46.675983 2887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c92a3764b8418e85d6d44e21a1ea77" candidate_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c140e639feee4ea9a7daf027f7d36e52" is_pre_election: true
I20260619 14:49:46.676062 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c92a3764b8418e85d6d44e21a1ea77" candidate_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2940371c6c4d84a6261123a28014d9" is_pre_election: true
I20260619 14:49:46.676290 3165 raft_consensus.cc:2454] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc15d47deb04ccba2237bf49c5b8a65 in term 0.
I20260619 14:49:46.676288 2887 raft_consensus.cc:2454] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc15d47deb04ccba2237bf49c5b8a65 in term 0.
I20260619 14:49:46.676731 2953 leader_election.cc:302] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e2940371c6c4d84a6261123a28014d9, 6cc15d47deb04ccba2237bf49c5b8a65; no voters:
I20260619 14:49:46.677007 3322 raft_consensus.cc:2781] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:46.677126 3322 raft_consensus.cc:482] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:46.677235 3322 raft_consensus.cc:3037] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:46.679308 3322 raft_consensus.cc:504] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.679852 3322 leader_election.cc:288] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [CANDIDATE]: Term 1 election: Requested vote from peers c140e639feee4ea9a7daf027f7d36e52 (127.0.241.1:36757), 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833)
I20260619 14:49:46.680280 2887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c92a3764b8418e85d6d44e21a1ea77" candidate_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c140e639feee4ea9a7daf027f7d36e52"
I20260619 14:49:46.680303 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c92a3764b8418e85d6d44e21a1ea77" candidate_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e2940371c6c4d84a6261123a28014d9"
I20260619 14:49:46.680526 2887 raft_consensus.cc:3037] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:46.680565 3165 raft_consensus.cc:3037] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:46.682761 3165 raft_consensus.cc:2454] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc15d47deb04ccba2237bf49c5b8a65 in term 1.
I20260619 14:49:46.683174 2953 leader_election.cc:302] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e2940371c6c4d84a6261123a28014d9, 6cc15d47deb04ccba2237bf49c5b8a65; no voters:
I20260619 14:49:46.683229 2887 raft_consensus.cc:2454] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc15d47deb04ccba2237bf49c5b8a65 in term 1.
I20260619 14:49:46.683470 3322 raft_consensus.cc:2781] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:46.683929 3322 raft_consensus.cc:686] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 LEADER]: Becoming Leader. State: Replica: 6cc15d47deb04ccba2237bf49c5b8a65, State: Running, Role: LEADER
I20260619 14:49:46.684348 3322 consensus_queue.cc:237] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } }
I20260619 14:49:46.687338 2723 catalog_manager.cc:5495] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2). New cstate: current_term: 1 leader_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cc15d47deb04ccba2237bf49c5b8a65" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 } health_report { overall_health: UNKNOWN } } }
I20260619 14:49:46.738020 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:46.726403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.0.241.2:34135)
0619 14:49:46.731851 (+ 5448us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:46.731856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:46.731874 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:49:46.731967 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:46.731971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:46.732047 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:46.732188 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:46.732599 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.733418 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.735772 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.736169 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.736259 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:46.736757 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:46.737367 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:46.737474 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:46.737624 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 14:49:46.737683 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5327,"thread_start_us":5196,"threads_started":1}
I20260619 14:49:46.753196 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:46.748190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:47777 (local address 127.0.241.1:36757)
0619 14:49:46.748405 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:46.748414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:46.748904 (+ 490us) server_negotiation.cc:407] Connection header received
0619 14:49:46.749097 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:46.749101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:46.749163 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:46.749236 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:46.749753 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.750418 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.751747 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.752163 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.752252 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:46.752626 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:46.752719 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:46.752781 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:46.752900 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 14:49:46.752956 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:49:46.753639 3337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:46.748172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:36757 (local address 127.0.241.2:47777)
0619 14:49:46.748789 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:46.748808 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:46.748971 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:46.749285 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:46.749290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:46.749303 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:49:46.749607 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:46.749615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.750531 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.750534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:46.751628 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:46.751639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.752307 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.752311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:46.752425 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:46.752473 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:49:46.752797 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:49:46.752804 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:49:46.752861 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:49:46.752965 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":327,"threads_started":1}
I20260619 14:49:46.761706 3165 raft_consensus.cc:1261] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Refusing update from remote peer 6cc15d47deb04ccba2237bf49c5b8a65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:46.762108 2887 raft_consensus.cc:1261] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Refusing update from remote peer 6cc15d47deb04ccba2237bf49c5b8a65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:46.762354 3332 consensus_queue.cc:1035] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e2940371c6c4d84a6261123a28014d9" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 42833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:46.762738 3322 consensus_queue.cc:1035] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c140e639feee4ea9a7daf027f7d36e52" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 36757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:46.789067 3334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:46.783779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:59071 (local address 127.0.241.2:34135)
0619 14:49:46.784085 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:46.784091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:46.784401 (+ 310us) server_negotiation.cc:407] Connection header received
0619 14:49:46.784569 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:46.784572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:46.784630 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:46.784701 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:46.785141 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.785943 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.787639 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.788135 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.788229 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:46.788559 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:46.788645 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:46.788709 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:46.788831 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 14:49:46.788885 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260619 14:49:46.789175 3343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:46.783671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:34135 (local address 127.0.241.1:59071)
0619 14:49:46.784292 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:46.784313 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:46.784449 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:46.784729 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:46.784732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:46.784741 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:49:46.785016 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:46.785023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.786060 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.786065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:46.787386 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:46.787403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:46.788283 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:46.788288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:46.788402 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:46.788437 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:49:46.788726 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:49:46.788734 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:49:46.788790 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 14:49:46.788886 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":329,"threads_started":1}
W20260619 14:49:46.790184 3289 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:49:48.396731 2725 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:48.396982 2725 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:48.400687 2725 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ce4f0c9b92b49febada095b584cf50e: Sending DeleteTablet for 3 replicas of tablet 97c92a3764b8418e85d6d44e21a1ea77
I20260619 14:49:48.401386 2862 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c92a3764b8418e85d6d44e21a1ea77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:48 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:52054
I20260619 14:49:48.401465 3001 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c92a3764b8418e85d6d44e21a1ea77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:48 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:36292
I20260619 14:49:48.401595 3139 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c92a3764b8418e85d6d44e21a1ea77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:48 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:33856
I20260619 14:49:48.402237 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 2796
I20260619 14:49:48.403939 3371 tablet_replica.cc:330] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9: stopping tablet replica
I20260619 14:49:48.404036 3370 tablet_replica.cc:330] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52: stopping tablet replica
I20260619 14:49:48.404189 3371 raft_consensus.cc:2229] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:48.404277 3370 raft_consensus.cc:2229] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:48.404458 3371 raft_consensus.cc:2258] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:48.404549 3370 raft_consensus.cc:2258] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:48.405587 3370 ts_tablet_manager.cc:1905] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:48.405941 3371 ts_tablet_manager.cc:1905] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:48.414034 3370 ts_tablet_manager.cc:1918] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:49:48.414285 3370 log.cc:1196] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/97c92a3764b8418e85d6d44e21a1ea77
I20260619 14:49:48.414619 3371 ts_tablet_manager.cc:1918] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:49:48.414781 3371 log.cc:1196] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/97c92a3764b8418e85d6d44e21a1ea77
I20260619 14:49:48.415023 3370 ts_tablet_manager.cc:1939] T 97c92a3764b8418e85d6d44e21a1ea77 P c140e639feee4ea9a7daf027f7d36e52: Deleting consensus metadata
I20260619 14:49:48.415326 3371 ts_tablet_manager.cc:1939] T 97c92a3764b8418e85d6d44e21a1ea77 P 4e2940371c6c4d84a6261123a28014d9: Deleting consensus metadata
I20260619 14:49:48.415884 3369 tablet_replica.cc:330] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65: stopping tablet replica
I20260619 14:49:48.416101 3369 raft_consensus.cc:2229] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:49:48.416636 2712 catalog_manager.cc:4841] TS c140e639feee4ea9a7daf027f7d36e52 (127.0.241.1:36757): tablet 97c92a3764b8418e85d6d44e21a1ea77 (table test-table [id=ade680d27e3a4395a81eaffbe448bf6a]) successfully deleted
I20260619 14:49:48.416685 3369 raft_consensus.cc:2258] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:48.417632 3369 ts_tablet_manager.cc:1905] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:48.417775 2710 catalog_manager.cc:4841] TS 4e2940371c6c4d84a6261123a28014d9 (127.0.241.3:42833): tablet 97c92a3764b8418e85d6d44e21a1ea77 (table test-table [id=ade680d27e3a4395a81eaffbe448bf6a]) successfully deleted
I20260619 14:49:48.423825 3369 ts_tablet_manager.cc:1918] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:49:48.423976 3369 log.cc:1196] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/97c92a3764b8418e85d6d44e21a1ea77
I20260619 14:49:48.424451 3369 ts_tablet_manager.cc:1939] T 97c92a3764b8418e85d6d44e21a1ea77 P 6cc15d47deb04ccba2237bf49c5b8a65: Deleting consensus metadata
I20260619 14:49:48.425611 2711 catalog_manager.cc:4841] TS 6cc15d47deb04ccba2237bf49c5b8a65 (127.0.241.2:34135): tablet 97c92a3764b8418e85d6d44e21a1ea77 (table test-table [id=ade680d27e3a4395a81eaffbe448bf6a]) successfully deleted
Jun 19 14:49:48 dist-test-slave-p3ml krb5kdc[2680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:49:48.431852 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:48.406296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.0.241.1:36757)
0619 14:49:48.416094 (+ 9798us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:48.416102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:48.416783 (+ 681us) server_negotiation.cc:407] Connection header received
0619 14:49:48.416842 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:48.416848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:48.416933 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:48.417060 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:48.418730 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:48.419488 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:48.421182 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:48.421591 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:48.421707 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:48.426899 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:48.426911 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:48.426933 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:48.426967 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:48.429641 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:48.430213 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:48.430219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:48.430221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:48.430342 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:48.430647 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:48.430650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:48.430652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:48.430957 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:48.431177 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:48.431393 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 14:49:48.431605 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9665,"thread_start_us":9519,"threads_started":1}
W20260619 14:49:48.432554 2927 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45824
W20260619 14:49:48.433168 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 2796: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:49:48.433336 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 2796
I20260619 14:49:48.459060 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 2936
Jun 19 14:49:48 dist-test-slave-p3ml krb5kdc[2680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:49:48.474184 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:48.459770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.0.241.2:34135)
0619 14:49:48.460269 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:48.460273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:48.460290 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:49:48.460341 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:48.460344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:48.460400 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:48.460492 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:48.461822 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:48.462618 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:48.463882 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:48.464297 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:48.464396 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:48.468478 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:48.468492 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:48.468507 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:48.468543 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:48.471531 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:48.472127 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:48.472134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:48.472136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:48.472281 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:48.472669 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:48.472676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:48.472679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:48.473042 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:48.473349 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:48.473625 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 14:49:48.473839 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
W20260619 14:49:48.474860 3066 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47978
W20260619 14:49:48.475675 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 2936: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:49:48.475824 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 2936
I20260619 14:49:48.497141 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 3075
Jun 19 14:49:48 dist-test-slave-p3ml krb5kdc[2680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:49:48.512414 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:48.497855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.0.241.3:42833)
0619 14:49:48.498386 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:48.498391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:48.498420 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:49:48.498473 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:48.498477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:48.498558 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:48.498681 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:48.500031 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:48.500777 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:48.501989 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:48.502415 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:48.502546 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:48.506840 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:48.506854 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:48.506869 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:48.506903 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:48.509622 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:48.510278 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:48.510285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:48.510287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:48.510414 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:48.510818 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:48.510823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:48.510825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:48.511210 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:48.511557 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:48.511844 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:49:48.512065 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":281,"threads_started":1}
W20260619 14:49:48.513051 3205 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51450
W20260619 14:49:48.513636 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 3075: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:49:48.513756 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 3075
I20260619 14:49:48.531206 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 2693
I20260619 14:49:48.532390 2765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:48.646154 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 2693
2026-06-19T14:49:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (10568 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:49:48 dist-test-slave-p3ml krb5kdc[3379](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:49:48 dist-test-slave-p3ml krb5kdc[3379](info): set up 2 sockets
Jun 19 14:49:48 dist-test-slave-p3ml krb5kdc[3379](info): commencing operation
krb5kdc: starting...
W20260619 14:49:49.739771 964 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:49:49 dist-test-slave-p3ml krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:49:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:49:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:49:49.923187 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:34147
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:34147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:50.249955 3392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:50.250269 3392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:50.250444 3392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:50.257561 3392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:50.257697 3392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:50.257794 3392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:50.257886 3392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:50.257964 3392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:50.267546 3392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:34147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:34147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:50.268843 3392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:50.270260 3392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:50.281965 3400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:50.282065 3397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:50.282043 3398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:50.282236 3392 server_base.cc:1034] running on GCE node
I20260619 14:49:50.283216 3392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:50.284597 3392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:50.285804 3392 hybrid_clock.cc:648] HybridClock initialized: now 1781880590285762 us; error 44 us; skew 500 ppm
I20260619 14:49:50.286135 3392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:50 dist-test-slave-p3ml krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:50.289356 3392 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:49:50.290781 3392 webserver.cc:460] Webserver started at http://127.0.241.62:45597/ using document root <none> and password file <none>
I20260619 14:49:50.291504 3392 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:50.291647 3392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:50.291954 3392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:50.294719 3392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "28f6fdf2dc804dfda6fe3266579838f2"
format_stamp: "Formatted at 2026-06-19 14:49:50 on dist-test-slave-p3ml"
server_key: "33f9cf62c0b81e397cdfb059dbe36d7a"
server_key_iv: "f7d472e20ffb6d5902b96695e57ead61"
server_key_version: "encryptionkey@0"
I20260619 14:49:50.295528 3392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "28f6fdf2dc804dfda6fe3266579838f2"
format_stamp: "Formatted at 2026-06-19 14:49:50 on dist-test-slave-p3ml"
server_key: "33f9cf62c0b81e397cdfb059dbe36d7a"
server_key_iv: "f7d472e20ffb6d5902b96695e57ead61"
server_key_version: "encryptionkey@0"
I20260619 14:49:50.300132 3392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:49:50.303236 3407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:50.304450 3392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:49:50.304618 3392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "28f6fdf2dc804dfda6fe3266579838f2"
format_stamp: "Formatted at 2026-06-19 14:49:50 on dist-test-slave-p3ml"
server_key: "33f9cf62c0b81e397cdfb059dbe36d7a"
server_key_iv: "f7d472e20ffb6d5902b96695e57ead61"
server_key_version: "encryptionkey@0"
I20260619 14:49:50.304796 3392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:50.331866 3392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:50.336472 3392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:50.336764 3392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:50.354362 3392 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:34147
I20260619 14:49:50.354393 3458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:34147 every 8 connection(s)
I20260619 14:49:50.355680 3392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:50.359090 3459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:50.359908 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 3392
I20260619 14:49:50.360116 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:49:50.360747 964 external_mini_cluster.cc:1428] Setting key 19d3e548ea92341356f59a73f1c94750
I20260619 14:49:50.372354 3459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2: Bootstrap starting.
Jun 19 14:49:50 dist-test-slave-p3ml krb5kdc[3379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:50.375787 3459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:50.376947 3459 log.cc:824] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:50.380007 3459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2: No bootstrap required, opened a new log
I20260619 14:49:50.382911 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:50.362600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.0.241.62:34147)
0619 14:49:50.363387 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:50.363397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:50.363478 (+ 81us) server_negotiation.cc:407] Connection header received
0619 14:49:50.365011 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:50.365033 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:50.365391 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:50.365711 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:50.367479 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:50.368721 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:50.370545 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:50.371172 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:50.371301 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:50.375712 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:50.375720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:50.375732 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:50.375763 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:50.379093 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:50.379895 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:50.379901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:50.379906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:50.380075 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:50.380486 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:50.380489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:50.380491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:50.380954 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:50.381163 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:50.381570 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 14:49:50.381874 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":373,"threads_started":1}
I20260619 14:49:50.387755 3459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f6fdf2dc804dfda6fe3266579838f2" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34147 } }
I20260619 14:49:50.388125 3459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:50.388243 3459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f6fdf2dc804dfda6fe3266579838f2, State: Initialized, Role: FOLLOWER
I20260619 14:49:50.388814 3459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f6fdf2dc804dfda6fe3266579838f2" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34147 } }
I20260619 14:49:50.389039 3459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:50.389153 3459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:50.389290 3459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:50.391304 3459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f6fdf2dc804dfda6fe3266579838f2" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34147 } }
I20260619 14:49:50.391769 3459 leader_election.cc:302] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28f6fdf2dc804dfda6fe3266579838f2; no voters:
I20260619 14:49:50.392279 3459 leader_election.cc:288] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:49:50.392423 3464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:50.393038 3464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [term 1 LEADER]: Becoming Leader. State: Replica: 28f6fdf2dc804dfda6fe3266579838f2, State: Running, Role: LEADER
I20260619 14:49:50.394084 3464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f6fdf2dc804dfda6fe3266579838f2" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34147 } }
I20260619 14:49:50.394332 3459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:50.396888 3466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28f6fdf2dc804dfda6fe3266579838f2. Latest consensus state: current_term: 1 leader_uuid: "28f6fdf2dc804dfda6fe3266579838f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f6fdf2dc804dfda6fe3266579838f2" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34147 } } }
I20260619 14:49:50.397415 3466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:50.396970 3465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28f6fdf2dc804dfda6fe3266579838f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f6fdf2dc804dfda6fe3266579838f2" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34147 } } }
I20260619 14:49:50.398070 3465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:50.399055 3473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:50.403880 3473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:50.412268 3473 catalog_manager.cc:1347] Generated new cluster ID: 00b24964446143ea82e07e08d85bf805
I20260619 14:49:50.412381 3473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:50.425463 3473 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:50.426626 3473 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:50.440855 3473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28f6fdf2dc804dfda6fe3266579838f2: Generated new TSK 0
I20260619 14:49:50.441781 3473 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:49:50.498378 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34147
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:50.814216 3485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:50.814565 3485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:50.814741 3485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:50.821738 3485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:50.821841 3485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:50.822019 3485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:49:50.831476 3485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:50.832684 3485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:50.834053 3485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:50.846884 3493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:50.847008 3491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:50.847008 3490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:50.848062 3485 server_base.cc:1034] running on GCE node
I20260619 14:49:50.848649 3485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:50.849655 3485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:50.850872 3485 hybrid_clock.cc:648] HybridClock initialized: now 1781880590850827 us; error 61 us; skew 500 ppm
I20260619 14:49:50.851188 3485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:50 dist-test-slave-p3ml krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:50.854254 3485 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:49:50.855794 3485 webserver.cc:460] Webserver started at http://127.0.241.1:35587/ using document root <none> and password file <none>
I20260619 14:49:50.856452 3485 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:50.856597 3485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:50.856895 3485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:50.859719 3485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "e68459ea09b44281b38accd938d01dc5"
format_stamp: "Formatted at 2026-06-19 14:49:50 on dist-test-slave-p3ml"
server_key: "82be50a5830b4f83ef04734afb288699"
server_key_iv: "54e4c39691d3bc650c257db0f4041373"
server_key_version: "encryptionkey@0"
I20260619 14:49:50.860481 3485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "e68459ea09b44281b38accd938d01dc5"
format_stamp: "Formatted at 2026-06-19 14:49:50 on dist-test-slave-p3ml"
server_key: "82be50a5830b4f83ef04734afb288699"
server_key_iv: "54e4c39691d3bc650c257db0f4041373"
server_key_version: "encryptionkey@0"
I20260619 14:49:50.865324 3485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:49:50.868611 3500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:50.869874 3485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:49:50.870064 3485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "e68459ea09b44281b38accd938d01dc5"
format_stamp: "Formatted at 2026-06-19 14:49:50 on dist-test-slave-p3ml"
server_key: "82be50a5830b4f83ef04734afb288699"
server_key_iv: "54e4c39691d3bc650c257db0f4041373"
server_key_version: "encryptionkey@0"
I20260619 14:49:50.870254 3485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:50.888473 3485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:50.892210 3485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:50.892484 3485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:50.893355 3485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:50.894701 3485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:50.894796 3485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:50.894937 3485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:50.895007 3485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:50.926642 3485 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:45659
I20260619 14:49:50.926724 3612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:45659 every 8 connection(s)
I20260619 14:49:50.928001 3485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:50.936506 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 3485
I20260619 14:49:50.936686 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:49:50.937312 964 external_mini_cluster.cc:1428] Setting key a8947a8fa92165a9c52e5960d102acb3
Jun 19 14:49:50 dist-test-slave-p3ml krb5kdc[3379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:50.947824 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:50.930381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:43905 (local address 127.0.241.62:34147)
0619 14:49:50.930781 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:50.930785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:50.931823 (+ 1038us) server_negotiation.cc:407] Connection header received
0619 14:49:50.933568 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:50.933573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:50.933682 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:50.933799 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:50.935210 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:50.935992 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:50.937690 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:50.938109 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:50.938240 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:50.941824 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:50.941833 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:50.941841 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:50.941885 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:50.944733 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:50.945618 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:50.945622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:50.945624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:50.945729 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:50.946172 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:50.946174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:50.946176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:50.946405 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:50.946506 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:50.947356 (+ 850us) server_negotiation.cc:299] Negotiation successful
0619 14:49:50.947583 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":204,"threads_started":1}
I20260619 14:49:50.949213 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:50.930945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34147 (local address 127.0.241.1:43905)
0619 14:49:50.931666 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:50.931701 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:50.933284 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:50.934030 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:50.934038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:50.934529 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:50.935046 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:50.935062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:50.936167 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:50.936171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:50.937554 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:50.937566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:50.938337 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:50.938344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:50.938576 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:50.939236 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:50.939261 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:50.941610 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:50.944869 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:50.944881 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:50.944899 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:50.945505 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:50.945826 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:50.945834 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:50.945840 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:50.946071 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:50.946525 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:50.946535 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:50.946958 (+ 423us) client_negotiation.cc:769] Sending connection context
0619 14:49:50.947291 (+ 333us) client_negotiation.cc:240] Negotiation successful
0619 14:49:50.947837 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":269,"threads_started":1}
I20260619 14:49:50.950592 3613 heartbeater.cc:347] Connected to a master server at 127.0.241.62:34147
I20260619 14:49:50.950861 3613 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:50.951618 3613 heartbeater.cc:511] Master 127.0.241.62:34147 requested a full tablet report, sending...
I20260619 14:49:50.953572 3424 ts_manager.cc:194] Registered new tserver with Master: e68459ea09b44281b38accd938d01dc5 (127.0.241.1:45659)
I20260619 14:49:50.954852 3424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:43905
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:49:50.979111 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34147
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:51.291052 3619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:51.291354 3619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:51.291560 3619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:51.298841 3619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:51.298964 3619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:51.299136 3619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:49:51.308543 3619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:51.309816 3619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:51.311088 3619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:51.323742 3624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:51.323760 3625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:51.323846 3627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:51.324651 3619 server_base.cc:1034] running on GCE node
I20260619 14:49:51.325233 3619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:51.326138 3619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:51.327353 3619 hybrid_clock.cc:648] HybridClock initialized: now 1781880591327309 us; error 62 us; skew 500 ppm
I20260619 14:49:51.327683 3619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:51 dist-test-slave-p3ml krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:51.333168 3619 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:49:51.334527 3619 webserver.cc:460] Webserver started at http://127.0.241.2:41475/ using document root <none> and password file <none>
I20260619 14:49:51.335229 3619 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:51.335325 3619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:51.335649 3619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:51.338308 3619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "b81ee8eeb72d469e99329b10a8374d4d"
format_stamp: "Formatted at 2026-06-19 14:49:51 on dist-test-slave-p3ml"
server_key: "57b54415ae773b1966da63f750c6b75a"
server_key_iv: "7a075d02729eac47d0e04fbed65a6d04"
server_key_version: "encryptionkey@0"
I20260619 14:49:51.339015 3619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "b81ee8eeb72d469e99329b10a8374d4d"
format_stamp: "Formatted at 2026-06-19 14:49:51 on dist-test-slave-p3ml"
server_key: "57b54415ae773b1966da63f750c6b75a"
server_key_iv: "7a075d02729eac47d0e04fbed65a6d04"
server_key_version: "encryptionkey@0"
I20260619 14:49:51.343657 3619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 14:49:51.346791 3634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:51.348029 3619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:51.348194 3619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "b81ee8eeb72d469e99329b10a8374d4d"
format_stamp: "Formatted at 2026-06-19 14:49:51 on dist-test-slave-p3ml"
server_key: "57b54415ae773b1966da63f750c6b75a"
server_key_iv: "7a075d02729eac47d0e04fbed65a6d04"
server_key_version: "encryptionkey@0"
I20260619 14:49:51.348376 3619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:51.362947 3619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:51.367812 3619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:51.368091 3619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:51.369040 3619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:51.370244 3619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:51.370321 3619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:51.370399 3619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:51.370460 3619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:51.400868 3619 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:43105
I20260619 14:49:51.400956 3746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:43105 every 8 connection(s)
I20260619 14:49:51.402133 3619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:49:51.405226 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 3619
I20260619 14:49:51.405393 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:49:51.405824 964 external_mini_cluster.cc:1428] Setting key 7d9f6e3f845d11334cf049dd7aec9d70
Jun 19 14:49:51 dist-test-slave-p3ml krb5kdc[3379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:51.425084 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:51.404826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:35083 (local address 127.0.241.62:34147)
0619 14:49:51.405051 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:51.405057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:51.406472 (+ 1415us) server_negotiation.cc:407] Connection header received
0619 14:49:51.408559 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:51.408563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:51.408665 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:51.408746 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:51.410844 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:51.411643 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:51.413974 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:51.414664 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:51.414813 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:51.418351 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:51.418361 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:51.418369 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:51.418413 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:51.422004 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:51.422679 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:51.422686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:51.422692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:51.422854 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:51.423237 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:51.423244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:51.423250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:51.423643 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:51.423804 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:51.424438 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 14:49:51.424683 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 14:49:51.425690 3749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:51.405439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34147 (local address 127.0.241.2:35083)
0619 14:49:51.406287 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:51.406333 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:51.408252 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:51.409012 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:51.409026 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:51.409852 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:51.410665 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:51.410694 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:51.411792 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:51.411801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:51.413831 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:51.413842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:51.414880 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:51.414885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:51.415076 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:51.415845 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:51.415866 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:51.418125 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:51.422146 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:51.422152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:51.422164 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:51.422559 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:51.422970 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:51.422973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:51.422975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:51.423129 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:51.423761 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:51.423766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:51.424052 (+ 286us) client_negotiation.cc:769] Sending connection context
0619 14:49:51.424284 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 14:49:51.424643 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":355,"threads_started":1}
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
I20260619 14:49:51.427101 3747 heartbeater.cc:347] Connected to a master server at 127.0.241.62:34147
I20260619 14:49:51.427403 3747 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:51.428323 3747 heartbeater.cc:511] Master 127.0.241.62:34147 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
I20260619 14:49:51.429877 3424 ts_manager.cc:194] Registered new tserver with Master: b81ee8eeb72d469e99329b10a8374d4d (127.0.241.2:43105)
I20260619 14:49:51.430676 3424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:35083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:49:51.450313 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34147
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:51.760975 3752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:51.761240 3752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:51.761332 3752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:51.768352 3752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:51.768486 3752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:51.768711 3752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:49:51.778317 3752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:51.779583 3752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:51.780885 3752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:51.793514 3758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:51.794050 3760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:51.794082 3757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:51.794495 3752 server_base.cc:1034] running on GCE node
I20260619 14:49:51.795140 3752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:51.796099 3752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:51.797299 3752 hybrid_clock.cc:648] HybridClock initialized: now 1781880591797268 us; error 29 us; skew 500 ppm
I20260619 14:49:51.797639 3752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:51 dist-test-slave-p3ml krb5kdc[3379](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:51.800776 3752 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:49:51.802138 3752 webserver.cc:460] Webserver started at http://127.0.241.3:46129/ using document root <none> and password file <none>
I20260619 14:49:51.802767 3752 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:51.802896 3752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:51.803242 3752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:51.806075 3752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "026553a940b448d886411c97e68b9c04"
format_stamp: "Formatted at 2026-06-19 14:49:51 on dist-test-slave-p3ml"
server_key: "db3492136d3f04515d843729dac5816b"
server_key_iv: "1457caa252be0c51ab6b55e43ee91332"
server_key_version: "encryptionkey@0"
I20260619 14:49:51.806843 3752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "026553a940b448d886411c97e68b9c04"
format_stamp: "Formatted at 2026-06-19 14:49:51 on dist-test-slave-p3ml"
server_key: "db3492136d3f04515d843729dac5816b"
server_key_iv: "1457caa252be0c51ab6b55e43ee91332"
server_key_version: "encryptionkey@0"
I20260619 14:49:51.811842 3752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 14:49:51.815032 3767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:51.816257 3752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:51.816422 3752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "026553a940b448d886411c97e68b9c04"
format_stamp: "Formatted at 2026-06-19 14:49:51 on dist-test-slave-p3ml"
server_key: "db3492136d3f04515d843729dac5816b"
server_key_iv: "1457caa252be0c51ab6b55e43ee91332"
server_key_version: "encryptionkey@0"
I20260619 14:49:51.816618 3752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:51.830431 3752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:51.833966 3752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:51.834246 3752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:51.835088 3752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:51.836484 3752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:51.836585 3752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:51.836737 3752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:51.836812 3752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:51.869208 3752 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:33597
I20260619 14:49:51.869247 3879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:33597 every 8 connection(s)
I20260619 14:49:51.870512 3752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:49:51.876456 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 3752
I20260619 14:49:51.876636 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:49:51.877058 964 external_mini_cluster.cc:1428] Setting key f11eb83947152e7b77ae1d03f0efab41
Jun 19 14:49:51 dist-test-slave-p3ml krb5kdc[3379](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:51.889990 3615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:51.872949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:45237 (local address 127.0.241.62:34147)
0619 14:49:51.873196 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:51.873202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:51.874407 (+ 1205us) server_negotiation.cc:407] Connection header received
0619 14:49:51.876301 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:51.876306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:51.876396 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:51.876508 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:51.878084 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:51.879006 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:51.880813 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:51.881224 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:51.881358 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:51.884634 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:51.884639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:51.884642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:51.884672 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:51.886943 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:51.887755 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:51.887760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:51.887764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:51.887885 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:51.888329 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:51.888335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:51.888338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:51.888604 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:51.888755 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:51.889415 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 14:49:51.889633 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:49:51.891100 3882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:51.873524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34147 (local address 127.0.241.3:45237)
0619 14:49:51.874245 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:51.874283 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:51.876035 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:51.876770 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:51.876781 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:51.877356 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:51.877913 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:51.877931 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:51.879157 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:51.879162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:51.880668 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:51.880680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:51.881442 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:51.881452 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:51.881698 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:51.882284 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:51.882305 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:51.884418 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:51.887100 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:51.887108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:51.887124 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:51.887632 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:51.888010 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:51.888016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:51.888019 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:51.888210 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:51.888732 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:51.888740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:51.889066 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 14:49:51.889320 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 14:49:51.889844 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":262,"threads_started":1}
I20260619 14:49:51.892541 3880 heartbeater.cc:347] Connected to a master server at 127.0.241.62:34147
I20260619 14:49:51.892853 3880 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:51.893548 3880 heartbeater.cc:511] Master 127.0.241.62:34147 requested a full tablet report, sending...
I20260619 14:49:51.895015 3424 ts_manager.cc:194] Registered new tserver with Master: 026553a940b448d886411c97e68b9c04 (127.0.241.3:33597)
I20260619 14:49:51.895913 3424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:45237
I20260619 14:49:51.904793 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:49:51.936574 3615 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:49:51.934201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.0.241.62:34147)
0619 14:49:51.934464 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:51.934471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:51.934791 (+ 320us) server_negotiation.cc:407] Connection header received
0619 14:49:51.934923 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:51.934926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:51.934988 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:51.935072 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:51.936310 (+ 1238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56660: BlockingRecv error: recv got EOF from 127.0.0.1:56660 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 14:49:51.940042 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 3485
W20260619 14:49:51.942819 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 3485: Not authorized: Client connection negotiation failed: client connection to 127.0.241.1:45659: server requires authentication, but client does not have Kerberos credentials available
I20260619 14:49:51.943005 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 3485
I20260619 14:49:51.955694 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 3619
W20260619 14:49:51.958411 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 3619: Not authorized: Client connection negotiation failed: client connection to 127.0.241.2:43105: server requires authentication, but client does not have Kerberos credentials available
I20260619 14:49:51.958576 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 3619
I20260619 14:49:51.974041 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 3752
W20260619 14:49:51.976756 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 3752: Not authorized: Client connection negotiation failed: client connection to 127.0.241.3:33597: server requires authentication, but client does not have Kerberos credentials available
I20260619 14:49:51.976902 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 3752
I20260619 14:49:51.991853 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 3392
I20260619 14:49:51.993080 3454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:52.093777 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 3392
2026-06-19T14:49:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3444 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:49:52 dist-test-slave-p3ml krb5kdc[3897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:49:52 dist-test-slave-p3ml krb5kdc[3897](info): set up 2 sockets
Jun 19 14:49:52 dist-test-slave-p3ml krb5kdc[3897](info): commencing operation
krb5kdc: starting...
W20260619 14:49:53.195693 964 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:49:53 dist-test-slave-p3ml krb5kdc[3897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:49:53.383319 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:44601
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:34495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:44601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:53.700282 3910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:53.700564 3910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:53.700649 3910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:53.707582 3910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:53.707674 3910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:53.707717 3910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:53.707772 3910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:53.707813 3910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:53.717096 3910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:34495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:44601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:53.718253 3910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:53.719400 3910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:53.730616 3915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:53.730719 3918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:53.730696 3916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:53.731750 3910 server_base.cc:1034] running on GCE node
I20260619 14:49:53.732419 3910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:53.733754 3910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:53.734962 3910 hybrid_clock.cc:648] HybridClock initialized: now 1781880593734921 us; error 39 us; skew 500 ppm
I20260619 14:49:53.735265 3910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:53 dist-test-slave-p3ml krb5kdc[3897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:53.738245 3910 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:49:53.739743 3910 webserver.cc:460] Webserver started at http://127.0.241.62:35841/ using document root <none> and password file <none>
I20260619 14:49:53.740384 3910 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:53.740607 3910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:53.740907 3910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:53.743852 3910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "c9f22ea12a10421da67905fb9d5c7884"
format_stamp: "Formatted at 2026-06-19 14:49:53 on dist-test-slave-p3ml"
server_key: "8112a7c5f1a1eebc81bf67c25f9733d4"
server_key_iv: "f7286ba3f38e0d7cd03a47b703008a4c"
server_key_version: "encryptionkey@0"
I20260619 14:49:53.744603 3910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "c9f22ea12a10421da67905fb9d5c7884"
format_stamp: "Formatted at 2026-06-19 14:49:53 on dist-test-slave-p3ml"
server_key: "8112a7c5f1a1eebc81bf67c25f9733d4"
server_key_iv: "f7286ba3f38e0d7cd03a47b703008a4c"
server_key_version: "encryptionkey@0"
I20260619 14:49:53.749614 3910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 14:49:53.752961 3925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:53.754243 3910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:49:53.754410 3910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "c9f22ea12a10421da67905fb9d5c7884"
format_stamp: "Formatted at 2026-06-19 14:49:53 on dist-test-slave-p3ml"
server_key: "8112a7c5f1a1eebc81bf67c25f9733d4"
server_key_iv: "f7286ba3f38e0d7cd03a47b703008a4c"
server_key_version: "encryptionkey@0"
I20260619 14:49:53.754594 3910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:53.770490 3910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:53.774292 3910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:53.774586 3910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:53.792575 3910 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:44601
I20260619 14:49:53.792594 3976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:44601 every 8 connection(s)
I20260619 14:49:53.793931 3910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:53.797575 3977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:53.800583 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 3910
I20260619 14:49:53.800808 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:49:53.801564 964 external_mini_cluster.cc:1428] Setting key ab388defdb8bc496ab954de875bd19fe
I20260619 14:49:53.809906 3977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884: Bootstrap starting.
I20260619 14:49:53.813621 3977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:53.814901 3977 log.cc:824] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:49:53 dist-test-slave-p3ml krb5kdc[3897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:53.818044 3977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884: No bootstrap required, opened a new log
I20260619 14:49:53.825281 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:53.803578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.0.241.62:44601)
0619 14:49:53.804352 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:53.804363 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:53.804403 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:49:53.805957 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:53.805978 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:53.806378 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:53.806696 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:53.808492 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:53.809744 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:53.811543 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:53.812202 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:53.812335 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:53.817069 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:53.817081 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:53.817100 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:53.817151 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:53.820962 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:53.821575 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:53.821583 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:53.821591 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:53.821801 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:53.822123 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:53.822132 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:53.822135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:53.822801 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:53.823090 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:53.823455 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 14:49:53.823900 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":387,"threads_started":1}
I20260619 14:49:53.828511 3977 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f22ea12a10421da67905fb9d5c7884" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 44601 } }
I20260619 14:49:53.828948 3977 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:53.829073 3977 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9f22ea12a10421da67905fb9d5c7884, State: Initialized, Role: FOLLOWER
I20260619 14:49:53.829327 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:49:53.829671 3977 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f22ea12a10421da67905fb9d5c7884" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 44601 } }
I20260619 14:49:53.829910 3977 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:53.830054 3977 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:53.830214 3977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:53.832407 3977 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f22ea12a10421da67905fb9d5c7884" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 44601 } }
I20260619 14:49:53.832880 3977 leader_election.cc:302] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9f22ea12a10421da67905fb9d5c7884; no voters:
I20260619 14:49:53.833463 3977 leader_election.cc:288] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:49:53.833595 3982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:53.839644 3982 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [term 1 LEADER]: Becoming Leader. State: Replica: c9f22ea12a10421da67905fb9d5c7884, State: Running, Role: LEADER
I20260619 14:49:53.840605 3982 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f22ea12a10421da67905fb9d5c7884" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 44601 } }
I20260619 14:49:53.841318 3977 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:53.843258 3984 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9f22ea12a10421da67905fb9d5c7884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f22ea12a10421da67905fb9d5c7884" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 44601 } } }
I20260619 14:49:53.843628 3984 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:53.844553 3983 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9f22ea12a10421da67905fb9d5c7884. Latest consensus state: current_term: 1 leader_uuid: "c9f22ea12a10421da67905fb9d5c7884" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9f22ea12a10421da67905fb9d5c7884" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 44601 } } }
I20260619 14:49:53.844913 3983 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:53.855484 3988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:53.861217 3988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:53.878350 3988 catalog_manager.cc:1347] Generated new cluster ID: d71bec868e9649089c9576ba8b3089f2
I20260619 14:49:53.878543 3988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:53.890619 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:53.880852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.0.241.62:44601)
0619 14:49:53.881039 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:53.881049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:53.884055 (+ 3006us) server_negotiation.cc:407] Connection header received
0619 14:49:53.884265 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:53.884271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:53.884345 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:53.884455 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:53.885442 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:53.886190 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:53.887496 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:53.887930 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:53.888071 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:53.889175 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:53.889179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:53.889183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:49:53.889799 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:53.889864 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:49:53.889887 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:53.890010 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:53.890154 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 14:49:53.890250 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:49:53.905989 3988 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:53.907358 3988 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:53.920533 3988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9f22ea12a10421da67905fb9d5c7884: Generated new TSK 0
I20260619 14:49:53.921407 3988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:49:53.928417 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 3910
I20260619 14:49:53.929945 3972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:54.019192 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 3910
2026-06-19T14:49:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1919 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:49:54 dist-test-slave-p3ml krb5kdc[4011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:49:54 dist-test-slave-p3ml krb5kdc[4011](info): set up 2 sockets
Jun 19 14:49:54 dist-test-slave-p3ml krb5kdc[4011](info): commencing operation
krb5kdc: starting...
W20260619 14:49:55.095749 964 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:49:55 dist-test-slave-p3ml krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:49:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:49:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:49:55.280448 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33445
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:33445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:55.592195 4024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:55.592468 4024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:55.592617 4024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:55.599874 4024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:55.599982 4024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:55.600118 4024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:55.600220 4024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:55.600302 4024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:55.609912 4024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:33445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:55.611236 4024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:55.612525 4024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:55.623997 4029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:55.624051 4030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:55.624090 4032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:55.625881 4024 server_base.cc:1034] running on GCE node
I20260619 14:49:55.626751 4024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:55.628259 4024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:55.629479 4024 hybrid_clock.cc:648] HybridClock initialized: now 1781880595629431 us; error 37 us; skew 500 ppm
I20260619 14:49:55.629848 4024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:55 dist-test-slave-p3ml krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:55.633002 4024 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:49:55.634510 4024 webserver.cc:460] Webserver started at http://127.0.241.62:34449/ using document root <none> and password file <none>
I20260619 14:49:55.635202 4024 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:55.635331 4024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:55.635661 4024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:55.638538 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "f99c4ee9da774f1d84ee98ae40e41768"
format_stamp: "Formatted at 2026-06-19 14:49:55 on dist-test-slave-p3ml"
server_key: "080cff3c64e2b1b895f927f4c78b9757"
server_key_iv: "3a1b8fdef86dc749bdbb838cad4ecdbd"
server_key_version: "encryptionkey@0"
I20260619 14:49:55.639281 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "f99c4ee9da774f1d84ee98ae40e41768"
format_stamp: "Formatted at 2026-06-19 14:49:55 on dist-test-slave-p3ml"
server_key: "080cff3c64e2b1b895f927f4c78b9757"
server_key_iv: "3a1b8fdef86dc749bdbb838cad4ecdbd"
server_key_version: "encryptionkey@0"
I20260619 14:49:55.644291 4024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:49:55.647702 4039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:55.648921 4024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:49:55.649101 4024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "f99c4ee9da774f1d84ee98ae40e41768"
format_stamp: "Formatted at 2026-06-19 14:49:55 on dist-test-slave-p3ml"
server_key: "080cff3c64e2b1b895f927f4c78b9757"
server_key_iv: "3a1b8fdef86dc749bdbb838cad4ecdbd"
server_key_version: "encryptionkey@0"
I20260619 14:49:55.649289 4024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:55.661753 4024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:55.665401 4024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:55.665674 4024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:55.683116 4024 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:33445
I20260619 14:49:55.683148 4090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:33445 every 8 connection(s)
I20260619 14:49:55.684499 4024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:55.687979 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 4024
I20260619 14:49:55.688146 4091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:55.688215 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:49:55.688987 964 external_mini_cluster.cc:1428] Setting key 2226d5164ec89b92bfd30ddeeda1bd7d
I20260619 14:49:55.701447 4091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768: Bootstrap starting.
Jun 19 14:49:55 dist-test-slave-p3ml krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:55.704828 4091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:55.705960 4091 log.cc:824] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:55.708757 4091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768: No bootstrap required, opened a new log
I20260619 14:49:55.711911 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:55.690925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.0.241.62:33445)
0619 14:49:55.691760 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:55.691770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:55.691808 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:49:55.694073 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:55.694093 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:55.694453 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:55.694760 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:55.696359 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:55.697560 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:55.699303 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:55.699978 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:55.700115 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:55.704574 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:55.704581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:55.704593 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:55.704629 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:55.708062 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:55.708697 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:55.708710 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:55.708716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:55.708913 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:55.709311 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:55.709316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:55.709319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:55.709797 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:55.710098 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:55.710526 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 14:49:55.710897 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":451,"threads_started":1}
I20260619 14:49:55.716341 4091 raft_consensus.cc:348] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99c4ee9da774f1d84ee98ae40e41768" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33445 } }
I20260619 14:49:55.716681 4091 raft_consensus.cc:374] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:55.716780 4091 raft_consensus.cc:729] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f99c4ee9da774f1d84ee98ae40e41768, State: Initialized, Role: FOLLOWER
I20260619 14:49:55.717370 4091 consensus_queue.cc:260] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99c4ee9da774f1d84ee98ae40e41768" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33445 } }
I20260619 14:49:55.717624 4091 raft_consensus.cc:388] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:55.717765 4091 raft_consensus.cc:482] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:55.717919 4091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:55.719894 4091 raft_consensus.cc:504] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99c4ee9da774f1d84ee98ae40e41768" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33445 } }
I20260619 14:49:55.720510 4091 leader_election.cc:302] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f99c4ee9da774f1d84ee98ae40e41768; no voters:
I20260619 14:49:55.721148 4091 leader_election.cc:288] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:49:55.721259 4096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:55.721923 4096 raft_consensus.cc:686] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [term 1 LEADER]: Becoming Leader. State: Replica: f99c4ee9da774f1d84ee98ae40e41768, State: Running, Role: LEADER
I20260619 14:49:55.722483 4096 consensus_queue.cc:237] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99c4ee9da774f1d84ee98ae40e41768" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33445 } }
I20260619 14:49:55.723268 4091 sys_catalog.cc:564] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:55.725240 4098 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f99c4ee9da774f1d84ee98ae40e41768. Latest consensus state: current_term: 1 leader_uuid: "f99c4ee9da774f1d84ee98ae40e41768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99c4ee9da774f1d84ee98ae40e41768" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33445 } } }
I20260619 14:49:55.725560 4098 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:55.730840 4097 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f99c4ee9da774f1d84ee98ae40e41768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99c4ee9da774f1d84ee98ae40e41768" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33445 } } }
I20260619 14:49:55.731179 4097 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768 [sys.catalog]: This master's current role is: LEADER
W20260619 14:49:55.735045 4112 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:49:55.735158 4112 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:49:55.735273 4104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:55.736843 4104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:55.744494 4104 catalog_manager.cc:1347] Generated new cluster ID: 6163e168efd14153af7e0238a65cff49
I20260619 14:49:55.744608 4104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:55.755810 4104 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:55.757077 4104 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:55.762461 4104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f99c4ee9da774f1d84ee98ae40e41768: Generated new TSK 0
I20260619 14:49:55.763217 4104 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:49:55.810258 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33445
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:56.121913 4117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:56.122171 4117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:56.122320 4117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:56.129695 4117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:56.129818 4117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:56.129995 4117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:49:56.139456 4117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:56.140650 4117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:56.141932 4117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:56.154767 4122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:56.154731 4123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:56.154743 4125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:56.155593 4117 server_base.cc:1034] running on GCE node
I20260619 14:49:56.156137 4117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:56.157061 4117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:56.158254 4117 hybrid_clock.cc:648] HybridClock initialized: now 1781880596158209 us; error 45 us; skew 500 ppm
I20260619 14:49:56.158519 4117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:56 dist-test-slave-p3ml krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:56.161573 4117 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:49:56.162964 4117 webserver.cc:460] Webserver started at http://127.0.241.1:33091/ using document root <none> and password file <none>
I20260619 14:49:56.163695 4117 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:56.163823 4117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:56.164120 4117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:56.166852 4117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "3f0fff3540f84285ab44b9d40b16844f"
format_stamp: "Formatted at 2026-06-19 14:49:56 on dist-test-slave-p3ml"
server_key: "90748a08a8ce6af74f3437aded1459ca"
server_key_iv: "07cc6e9ceda648401b346c3a52d0c1f2"
server_key_version: "encryptionkey@0"
I20260619 14:49:56.167675 4117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "3f0fff3540f84285ab44b9d40b16844f"
format_stamp: "Formatted at 2026-06-19 14:49:56 on dist-test-slave-p3ml"
server_key: "90748a08a8ce6af74f3437aded1459ca"
server_key_iv: "07cc6e9ceda648401b346c3a52d0c1f2"
server_key_version: "encryptionkey@0"
I20260619 14:49:56.172572 4117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:49:56.175872 4132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:56.177156 4117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 14:49:56.177320 4117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "3f0fff3540f84285ab44b9d40b16844f"
format_stamp: "Formatted at 2026-06-19 14:49:56 on dist-test-slave-p3ml"
server_key: "90748a08a8ce6af74f3437aded1459ca"
server_key_iv: "07cc6e9ceda648401b346c3a52d0c1f2"
server_key_version: "encryptionkey@0"
I20260619 14:49:56.177502 4117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:56.200387 4117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:56.204212 4117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:56.204499 4117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:56.205417 4117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:56.206730 4117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:56.206831 4117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:56.206969 4117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:56.207046 4117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:56.239504 4117 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:33507
I20260619 14:49:56.239619 4244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:33507 every 8 connection(s)
I20260619 14:49:56.240762 4117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:56.247040 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 4117
I20260619 14:49:56.247255 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:49:56.247923 964 external_mini_cluster.cc:1428] Setting key ba5ea02282e440dd651e1d87c73e73e0
Jun 19 14:49:56 dist-test-slave-p3ml krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:56.262812 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:56.243262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:38283 (local address 127.0.241.62:33445)
0619 14:49:56.243706 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:56.243711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:56.244700 (+ 989us) server_negotiation.cc:407] Connection header received
0619 14:49:56.246445 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:56.246451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:56.246532 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:56.246674 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:56.248525 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:56.249508 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:56.251958 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:56.252373 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:56.252474 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:56.256771 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:56.256779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:56.256788 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:56.256819 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:56.259330 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:56.260295 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:56.260299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:56.260301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:56.260406 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:56.260895 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:56.260898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:56.260900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:56.261153 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:56.261254 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:56.262234 (+ 980us) server_negotiation.cc:299] Negotiation successful
0619 14:49:56.262408 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":257,"threads_started":1}
I20260619 14:49:56.264232 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:56.243865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33445 (local address 127.0.241.1:38283)
0619 14:49:56.244549 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:56.244586 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:56.246191 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:56.247005 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:56.247033 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:56.247713 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:56.248336 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:56.248353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:56.249662 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:56.249667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:56.251811 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:56.251830 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:56.252631 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:56.252641 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:56.252967 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:56.253699 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:56.253732 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:56.256511 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:56.259523 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:56.259536 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:56.259556 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:56.260176 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:56.260525 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:56.260533 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:56.260540 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:56.260787 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:56.261276 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:56.261286 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:56.261764 (+ 478us) client_negotiation.cc:769] Sending connection context
0619 14:49:56.262077 (+ 313us) client_negotiation.cc:240] Negotiation successful
0619 14:49:56.262679 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":269,"threads_started":1}
I20260619 14:49:56.265834 4245 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33445
I20260619 14:49:56.266136 4245 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:56.266824 4245 heartbeater.cc:511] Master 127.0.241.62:33445 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
I20260619 14:49:56.268884 4056 ts_manager.cc:194] Registered new tserver with Master: 3f0fff3540f84285ab44b9d40b16844f (127.0.241.1:33507)
I20260619 14:49:56.270421 4056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:38283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:49:56.294034 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33445
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:56.618928 4251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:56.619206 4251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:56.619304 4251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:56.626722 4251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:56.626825 4251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:56.626940 4251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:49:56.636592 4251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:56.637734 4251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:56.638969 4251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:56.652837 4256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:56.652792 4257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:56.653308 4259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:56.653784 4251 server_base.cc:1034] running on GCE node
I20260619 14:49:56.654440 4251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:56.655398 4251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:56.656649 4251 hybrid_clock.cc:648] HybridClock initialized: now 1781880596656610 us; error 31 us; skew 500 ppm
I20260619 14:49:56.656955 4251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:56 dist-test-slave-p3ml krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:56.664531 4251 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:49:56.665908 4251 webserver.cc:460] Webserver started at http://127.0.241.2:40987/ using document root <none> and password file <none>
I20260619 14:49:56.666554 4251 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:56.666690 4251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:56.666988 4251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:56.669694 4251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "66159def9caa49a0a039f9161d1417f3"
format_stamp: "Formatted at 2026-06-19 14:49:56 on dist-test-slave-p3ml"
server_key: "b6caab139f4791632f5a34742f981440"
server_key_iv: "cd1e9c7b787686935aa2af92dd4b1496"
server_key_version: "encryptionkey@0"
I20260619 14:49:56.670452 4251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "66159def9caa49a0a039f9161d1417f3"
format_stamp: "Formatted at 2026-06-19 14:49:56 on dist-test-slave-p3ml"
server_key: "b6caab139f4791632f5a34742f981440"
server_key_iv: "cd1e9c7b787686935aa2af92dd4b1496"
server_key_version: "encryptionkey@0"
I20260619 14:49:56.675360 4251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 14:49:56.678696 4266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:56.680054 4251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:56.680225 4251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "66159def9caa49a0a039f9161d1417f3"
format_stamp: "Formatted at 2026-06-19 14:49:56 on dist-test-slave-p3ml"
server_key: "b6caab139f4791632f5a34742f981440"
server_key_iv: "cd1e9c7b787686935aa2af92dd4b1496"
server_key_version: "encryptionkey@0"
I20260619 14:49:56.680411 4251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:56.701648 4251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:56.706029 4251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:56.706264 4251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:56.707082 4251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:56.708382 4251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:56.708451 4251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:56.708532 4251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:56.708577 4251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:56.740688 4251 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:45249
I20260619 14:49:56.740782 4378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:45249 every 8 connection(s)
I20260619 14:49:56.741978 4251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:49:56.750972 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 4251
I20260619 14:49:56.751179 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:49:56.751679 964 external_mini_cluster.cc:1428] Setting key 9ce08139b56dbb4905701e5e05b23e6a
Jun 19 14:49:56 dist-test-slave-p3ml krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:56.761598 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:56.744443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:47015 (local address 127.0.241.62:33445)
0619 14:49:56.744638 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:56.744642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:56.745748 (+ 1106us) server_negotiation.cc:407] Connection header received
0619 14:49:56.747416 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:56.747420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:56.747542 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:56.747667 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:56.749088 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:56.749951 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:56.751605 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:56.752078 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:56.752216 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:56.756146 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:56.756151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:56.756155 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:56.756183 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:56.758682 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:56.759389 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:56.759397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:56.759403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:56.759589 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:56.760002 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:56.760009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:56.760012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:56.760275 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:56.760428 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:56.761063 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 14:49:56.761271 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:49:56.762506 4381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:56.744912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33445 (local address 127.0.241.2:47015)
0619 14:49:56.745590 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:56.745627 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:56.747170 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:56.747907 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:56.747915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:56.748403 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:56.748934 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:56.748951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:56.750085 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:56.750089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:56.751466 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:56.751482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:56.752318 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:56.752324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:56.752562 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:56.753329 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:56.753370 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:56.755926 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:56.758816 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:56.758824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:56.758840 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:56.759276 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:56.759714 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:56.759720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:56.759723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:56.759896 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:56.760395 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:56.760403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:56.760724 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 14:49:56.760981 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 14:49:56.761471 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":266,"threads_started":1}
I20260619 14:49:56.764070 4379 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33445
I20260619 14:49:56.764357 4379 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:56.765048 4379 heartbeater.cc:511] Master 127.0.241.62:33445 requested a full tablet report, sending...
I20260619 14:49:56.766575 4056 ts_manager.cc:194] Registered new tserver with Master: 66159def9caa49a0a039f9161d1417f3 (127.0.241.2:45249)
I20260619 14:49:56.767746 4056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:47015
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:49:56.794186 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33445
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:57.105225 4384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:57.105490 4384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:57.105584 4384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:57.112524 4384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:57.112622 4384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:57.112730 4384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:49:57.122548 4384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:57.123766 4384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:57.124938 4384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:57.137593 4389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:57.137593 4390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:57.137523 4392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:57.138387 4384 server_base.cc:1034] running on GCE node
I20260619 14:49:57.138936 4384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:57.139876 4384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:57.141076 4384 hybrid_clock.cc:648] HybridClock initialized: now 1781880597141038 us; error 34 us; skew 500 ppm
I20260619 14:49:57.141415 4384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:57 dist-test-slave-p3ml krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:57.144431 4384 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:49:57.145860 4384 webserver.cc:460] Webserver started at http://127.0.241.3:42317/ using document root <none> and password file <none>
I20260619 14:49:57.146514 4384 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:57.146646 4384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:57.146948 4384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:57.149904 4384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "aeac236ac4df42b286d75823bda20016"
format_stamp: "Formatted at 2026-06-19 14:49:57 on dist-test-slave-p3ml"
server_key: "ae21e5fdf74541e0536b8714ef533135"
server_key_iv: "ba9bf5f2bc0d83cbc949cd34175d8b2a"
server_key_version: "encryptionkey@0"
I20260619 14:49:57.150695 4384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "aeac236ac4df42b286d75823bda20016"
format_stamp: "Formatted at 2026-06-19 14:49:57 on dist-test-slave-p3ml"
server_key: "ae21e5fdf74541e0536b8714ef533135"
server_key_iv: "ba9bf5f2bc0d83cbc949cd34175d8b2a"
server_key_version: "encryptionkey@0"
I20260619 14:49:57.156054 4384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 14:49:57.159247 4399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:57.160612 4384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:49:57.160781 4384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "aeac236ac4df42b286d75823bda20016"
format_stamp: "Formatted at 2026-06-19 14:49:57 on dist-test-slave-p3ml"
server_key: "ae21e5fdf74541e0536b8714ef533135"
server_key_iv: "ba9bf5f2bc0d83cbc949cd34175d8b2a"
server_key_version: "encryptionkey@0"
I20260619 14:49:57.160966 4384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:57.183601 4384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:57.187577 4384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:57.187925 4384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:57.188779 4384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:57.190287 4384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:57.190423 4384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:57.190591 4384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:57.190671 4384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:57.223706 4384 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:40297
I20260619 14:49:57.223822 4511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:40297 every 8 connection(s)
I20260619 14:49:57.225051 4384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:49:57.230871 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 4384
I20260619 14:49:57.231069 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:49:57.231582 964 external_mini_cluster.cc:1428] Setting key 840bcfd7dd6f6bca7941ad3ec5791b1f
Jun 19 14:49:57 dist-test-slave-p3ml krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:57.244163 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:57.227418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:39673 (local address 127.0.241.62:33445)
0619 14:49:57.227654 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:57.227658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:57.228712 (+ 1054us) server_negotiation.cc:407] Connection header received
0619 14:49:57.230447 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:57.230452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:57.230593 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:57.230701 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:57.232408 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:57.233448 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:57.235029 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:57.235479 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:57.235596 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:57.238692 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:57.238696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:57.238700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:57.238729 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:57.241271 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:57.242076 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:57.242080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:57.242082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:57.242183 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:57.242637 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:57.242640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:57.242642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:57.242881 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:57.242986 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:57.243777 (+ 791us) server_negotiation.cc:299] Negotiation successful
0619 14:49:57.243918 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 14:49:57.245407 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:57.227868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33445 (local address 127.0.241.3:39673)
0619 14:49:57.228535 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:57.228582 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:57.230198 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:57.230926 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:57.230959 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:57.231609 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:57.232226 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:57.232249 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:57.233618 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:57.233622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:57.234908 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:57.234919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:57.235674 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:57.235679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:57.235857 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:57.236458 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:57.236492 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:57.238493 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:57.241405 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:57.241414 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:57.241428 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:57.241973 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:57.242288 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:57.242296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:57.242302 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:57.242543 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:57.242994 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:57.243001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:57.243372 (+ 371us) client_negotiation.cc:769] Sending connection context
0619 14:49:57.243705 (+ 333us) client_negotiation.cc:240] Negotiation successful
0619 14:49:57.244180 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":278,"threads_started":1}
I20260619 14:49:57.246706 4512 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33445
I20260619 14:49:57.247027 4512 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:57.247758 4512 heartbeater.cc:511] Master 127.0.241.62:33445 requested a full tablet report, sending...
I20260619 14:49:57.249229 4055 ts_manager.cc:194] Registered new tserver with Master: aeac236ac4df42b286d75823bda20016 (127.0.241.3:40297)
I20260619 14:49:57.250200 4055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:39673
I20260619 14:49:57.259050 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:49:57.274165 4245 heartbeater.cc:503] Master 127.0.241.62:33445 was elected leader, sending a full tablet report...
Jun 19 14:49:57 dist-test-slave-p3ml krb5kdc[4011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880597, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 14:49:57 dist-test-slave-p3ml krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880597, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:57.306355 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:57.292544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.0.241.62:33445)
0619 14:49:57.292771 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:57.292779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:57.293152 (+ 373us) server_negotiation.cc:407] Connection header received
0619 14:49:57.293368 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:57.293373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:57.293433 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:57.293539 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:57.294880 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:57.295680 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:57.296901 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:57.297324 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:57.297451 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:57.301259 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:57.301264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:57.301267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:57.301299 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:57.303852 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:57.304471 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:57.304478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:57.304482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:57.304624 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:57.304976 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:57.304983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:57.304987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:57.305291 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:57.305530 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:57.305817 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:49:57.306020 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 14:49:57.306924 4055 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58480
I20260619 14:49:57.308386 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 4117
Jun 19 14:49:57 dist-test-slave-p3ml krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880597, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:49:57.322724 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:57.309444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.0.241.1:33507)
0619 14:49:57.310061 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:57.310068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:57.310091 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:49:57.310147 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:57.310153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:57.310344 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:57.310476 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:57.311530 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:57.312497 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:57.313605 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:57.314145 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:57.314234 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:57.317891 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:57.317902 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:57.317916 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:57.317946 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:57.320550 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:57.321146 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:57.321152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:57.321154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:57.321263 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:57.321549 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:57.321552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:57.321554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:57.321859 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:57.322067 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:57.322279 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 14:49:57.322456 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":319,"threads_started":1}
W20260619 14:49:57.324024 4240 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36088
W20260619 14:49:57.325083 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 4117: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:49:57.325228 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 4117
I20260619 14:49:57.340622 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 4251
Jun 19 14:49:57 dist-test-slave-p3ml krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880597, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:49:57.355249 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:57.341411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.0.241.2:45249)
0619 14:49:57.341938 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:57.341945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:57.341967 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:49:57.342027 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:57.342033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:57.342221 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:57.342346 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:57.343489 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:57.344507 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:57.345650 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:57.346202 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:57.346295 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:57.350162 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:57.350171 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:57.350184 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:57.350215 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:57.352854 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:57.353400 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:57.353407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:57.353409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:57.353579 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:57.353921 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:57.353924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:57.353926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:57.354215 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:57.354418 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:57.354705 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:49:57.354914 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":285,"threads_started":1}
W20260619 14:49:57.356454 4374 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39994
W20260619 14:49:57.357415 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 4251: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:49:57.357586 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 4251
I20260619 14:49:57.372922 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 4384
Jun 19 14:49:57 dist-test-slave-p3ml krb5kdc[4011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880597, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:49:57.387883 4525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:57.373790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.0.241.3:40297)
0619 14:49:57.374447 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:57.374454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:57.374481 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:49:57.374534 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:57.374540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:57.374730 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:57.374845 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:57.376057 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:57.377065 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:57.378184 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:57.378695 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:57.378793 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:57.382714 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:57.382723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:57.382735 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:57.382768 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:57.385494 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:57.386083 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:57.386095 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:57.386101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:57.386269 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:57.386589 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:57.386592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:57.386594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:57.386897 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:57.387119 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:57.387295 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 14:49:57.387534 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":373,"threads_started":1}
W20260619 14:49:57.389045 4507 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48680
W20260619 14:49:57.390144 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 4384: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:49:57.390300 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 4384
I20260619 14:49:57.405386 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 4024
I20260619 14:49:57.406479 4086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:49:57.507198 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 4024
2026-06-19T14:49:57Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3493 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:49:57 dist-test-slave-p3ml krb5kdc[4529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:49:57 dist-test-slave-p3ml krb5kdc[4529](info): set up 2 sockets
Jun 19 14:49:57 dist-test-slave-p3ml krb5kdc[4529](info): commencing operation
krb5kdc: starting...
W20260619 14:49:58.587862 964 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:49:58 dist-test-slave-p3ml krb5kdc[4529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:49:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:49:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:49:58.768623 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:40081
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:40081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:59.072405 4542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:59.072690 4542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:59.072815 4542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:59.079608 4542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:59.079754 4542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:59.079839 4542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:59.079933 4542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:59.080013 4542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:59.089569 4542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:40081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:40081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:59.090852 4542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:59.092270 4542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:59.103513 4547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:59.103514 4548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:59.103546 4550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:59.104519 4542 server_base.cc:1034] running on GCE node
I20260619 14:49:59.105299 4542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:59.106573 4542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:59.107788 4542 hybrid_clock.cc:648] HybridClock initialized: now 1781880599107751 us; error 43 us; skew 500 ppm
I20260619 14:49:59.108109 4542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:59 dist-test-slave-p3ml krb5kdc[4529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:59.111022 4542 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:49:59.112555 4542 webserver.cc:460] Webserver started at http://127.0.241.62:45689/ using document root <none> and password file <none>
I20260619 14:49:59.113196 4542 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:59.113335 4542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:59.113634 4542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:59.116456 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "d9c8f0c9d3af4bc3a4586fecde13f3cd"
format_stamp: "Formatted at 2026-06-19 14:49:59 on dist-test-slave-p3ml"
server_key: "7b8c6648fe4f3861e99776aa4970de90"
server_key_iv: "f0eeb3fc41fb9640c538e54396cb4a38"
server_key_version: "encryptionkey@0"
I20260619 14:49:59.117182 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "d9c8f0c9d3af4bc3a4586fecde13f3cd"
format_stamp: "Formatted at 2026-06-19 14:49:59 on dist-test-slave-p3ml"
server_key: "7b8c6648fe4f3861e99776aa4970de90"
server_key_iv: "f0eeb3fc41fb9640c538e54396cb4a38"
server_key_version: "encryptionkey@0"
I20260619 14:49:59.121889 4542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 14:49:59.125084 4557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:59.126274 4542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:49:59.126469 4542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "d9c8f0c9d3af4bc3a4586fecde13f3cd"
format_stamp: "Formatted at 2026-06-19 14:49:59 on dist-test-slave-p3ml"
server_key: "7b8c6648fe4f3861e99776aa4970de90"
server_key_iv: "f0eeb3fc41fb9640c538e54396cb4a38"
server_key_version: "encryptionkey@0"
I20260619 14:49:59.126652 4542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:59.145268 4542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:59.149061 4542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:59.149330 4542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:59.166930 4542 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:40081
I20260619 14:49:59.166963 4608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:40081 every 8 connection(s)
I20260619 14:49:59.168239 4542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:59.171803 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:59.175241 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 4542
I20260619 14:49:59.175474 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:49:59.176277 964 external_mini_cluster.cc:1428] Setting key 51a64c62d465124bc3bd5c80635af4ba
I20260619 14:49:59.183822 4609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd: Bootstrap starting.
I20260619 14:49:59.186549 4609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:59.187685 4609 log.cc:824] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:49:59 dist-test-slave-p3ml krb5kdc[4529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:59.190490 4609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd: No bootstrap required, opened a new log
I20260619 14:49:59.198385 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:59.178268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.0.241.62:40081)
0619 14:49:59.179005 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:59.179016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:59.179063 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:49:59.180645 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:59.180666 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:59.181031 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:59.181428 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:59.182852 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:59.184231 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:59.185415 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:59.186043 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:59.186369 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:59.190823 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:59.190830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:59.190843 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:59.190884 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:59.194064 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:59.194734 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:59.194741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:59.194746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:59.194912 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:59.195535 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:59.195542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:59.195546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:59.196122 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:59.196438 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:59.196955 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 14:49:59.197378 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":351,"threads_started":1}
I20260619 14:49:59.198961 4609 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c8f0c9d3af4bc3a4586fecde13f3cd" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 40081 } }
I20260619 14:49:59.199304 4609 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:59.199420 4609 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c8f0c9d3af4bc3a4586fecde13f3cd, State: Initialized, Role: FOLLOWER
I20260619 14:49:59.200098 4609 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c8f0c9d3af4bc3a4586fecde13f3cd" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 40081 } }
I20260619 14:49:59.200405 4609 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:59.200570 4609 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:59.200722 4609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:59.203624 4609 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c8f0c9d3af4bc3a4586fecde13f3cd" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 40081 } }
I20260619 14:49:59.204046 4609 leader_election.cc:302] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9c8f0c9d3af4bc3a4586fecde13f3cd; no voters:
I20260619 14:49:59.204617 4609 leader_election.cc:288] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:49:59.204718 4614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:59.206213 4614 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [term 1 LEADER]: Becoming Leader. State: Replica: d9c8f0c9d3af4bc3a4586fecde13f3cd, State: Running, Role: LEADER
I20260619 14:49:59.206786 4614 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c8f0c9d3af4bc3a4586fecde13f3cd" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 40081 } }
I20260619 14:49:59.206983 4609 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:59.216586 4616 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9c8f0c9d3af4bc3a4586fecde13f3cd. Latest consensus state: current_term: 1 leader_uuid: "d9c8f0c9d3af4bc3a4586fecde13f3cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c8f0c9d3af4bc3a4586fecde13f3cd" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 40081 } } }
I20260619 14:49:59.216954 4616 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:59.216892 4615 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9c8f0c9d3af4bc3a4586fecde13f3cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c8f0c9d3af4bc3a4586fecde13f3cd" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 40081 } } }
I20260619 14:49:59.217212 4615 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:59.217792 4630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:59.221829 4630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:59.229420 4630 catalog_manager.cc:1347] Generated new cluster ID: 687a4e00b1fa4234a18e3ef7b8c68fcd
I20260619 14:49:59.229527 4630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:59.240787 4630 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:59.242002 4630 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:59.256325 4630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9c8f0c9d3af4bc3a4586fecde13f3cd: Generated new TSK 0
I20260619 14:49:59.257376 4630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:49:59.315110 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:40081
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:59.630967 4636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:59.631299 4636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:59.631492 4636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:59.638710 4636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:59.638861 4636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:59.639035 4636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:49:59.648897 4636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:40081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:49:59.650238 4636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:59.651638 4636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:59.664809 4641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:59.664882 4642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:59.665017 4644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:59.666018 4636 server_base.cc:1034] running on GCE node
I20260619 14:49:59.666649 4636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:59.667614 4636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:59.668835 4636 hybrid_clock.cc:648] HybridClock initialized: now 1781880599668785 us; error 45 us; skew 500 ppm
I20260619 14:49:59.669152 4636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:49:59 dist-test-slave-p3ml krb5kdc[4529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:59.672271 4636 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:49:59.673645 4636 webserver.cc:460] Webserver started at http://127.0.241.1:40275/ using document root <none> and password file <none>
I20260619 14:49:59.674336 4636 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:59.674479 4636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:59.674772 4636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:59.677685 4636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "a2d7555ef56c4827b23e54a535ace196"
format_stamp: "Formatted at 2026-06-19 14:49:59 on dist-test-slave-p3ml"
server_key: "ec5ead5778f94a310e978b6c5f7746ec"
server_key_iv: "c83472b8f6a631e54886be410c5dbca8"
server_key_version: "encryptionkey@0"
I20260619 14:49:59.678486 4636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "a2d7555ef56c4827b23e54a535ace196"
format_stamp: "Formatted at 2026-06-19 14:49:59 on dist-test-slave-p3ml"
server_key: "ec5ead5778f94a310e978b6c5f7746ec"
server_key_iv: "c83472b8f6a631e54886be410c5dbca8"
server_key_version: "encryptionkey@0"
I20260619 14:49:59.683815 4636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 14:49:59.687060 4651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:59.688380 4636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:59.688553 4636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "a2d7555ef56c4827b23e54a535ace196"
format_stamp: "Formatted at 2026-06-19 14:49:59 on dist-test-slave-p3ml"
server_key: "ec5ead5778f94a310e978b6c5f7746ec"
server_key_iv: "c83472b8f6a631e54886be410c5dbca8"
server_key_version: "encryptionkey@0"
I20260619 14:49:59.688764 4636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:59.703302 4636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:59.707267 4636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:59.707602 4636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:59.708726 4636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:59.710175 4636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:59.710268 4636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:59.710428 4636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:59.710489 4636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:59.743680 4636 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:34003
I20260619 14:49:59.743726 4763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:34003 every 8 connection(s)
I20260619 14:49:59.745025 4636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:59.754021 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 4636
I20260619 14:49:59.754220 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:49:59.754863 964 external_mini_cluster.cc:1428] Setting key c674877d52d3601b24bda146755d6cc6
Jun 19 14:49:59 dist-test-slave-p3ml krb5kdc[4529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:49:59.766790 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:59.747593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:36769 (local address 127.0.241.62:40081)
0619 14:49:59.748029 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:59.748033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:59.748979 (+ 946us) server_negotiation.cc:407] Connection header received
0619 14:49:59.750813 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:59.750818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:59.750921 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:59.751072 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:59.752734 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:59.753476 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:59.755332 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:59.756068 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:59.756216 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:59.760239 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:59.760249 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:59.760256 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:59.760303 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:59.763515 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:59.764484 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:59.764487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:59.764490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:59.764630 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:59.765104 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:59.765107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:59.765109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:59.765364 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:59.765471 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:59.766349 (+ 878us) server_negotiation.cc:299] Negotiation successful
0619 14:49:59.766512 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":245,"threads_started":1}
I20260619 14:49:59.768404 4767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:59.748083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:40081 (local address 127.0.241.1:36769)
0619 14:49:59.748797 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:59.748841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:59.750517 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:59.751329 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:59.751338 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:59.751934 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:59.752536 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:59.752553 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:59.753670 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:59.753674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:59.755178 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:59.755192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:59.756327 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:59.756334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:59.756664 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:59.757439 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:59.757462 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:59.759987 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:59.763669 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:59.763681 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:59.763700 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:59.764364 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:59.764733 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:59.764741 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:59.764748 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:59.764995 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:59.765483 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:59.765494 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:59.765960 (+ 466us) client_negotiation.cc:769] Sending connection context
0619 14:49:59.766275 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 14:49:59.766878 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":280,"threads_started":1}
I20260619 14:49:59.769940 4764 heartbeater.cc:347] Connected to a master server at 127.0.241.62:40081
I20260619 14:49:59.770246 4764 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:59.770957 4764 heartbeater.cc:511] Master 127.0.241.62:40081 requested a full tablet report, sending...
I20260619 14:49:59.773077 4574 ts_manager.cc:194] Registered new tserver with Master: a2d7555ef56c4827b23e54a535ace196 (127.0.241.1:34003)
I20260619 14:49:59.774456 4574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:36769
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:49:59.798327 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:40081
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:00.104691 4770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:00.104995 4770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:00.105156 4770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:00.112658 4770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:00.112805 4770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:00.112964 4770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:50:00.122149 4770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:40081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:00.123328 4770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:00.124609 4770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:00.138500 4778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:00.138556 4776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:00.138517 4775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:00.139643 4770 server_base.cc:1034] running on GCE node
I20260619 14:50:00.140192 4770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:00.141006 4770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:00.142230 4770 hybrid_clock.cc:648] HybridClock initialized: now 1781880600142185 us; error 38 us; skew 500 ppm
I20260619 14:50:00.142509 4770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:00 dist-test-slave-p3ml krb5kdc[4529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:00.145571 4770 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:50:00.146981 4770 webserver.cc:460] Webserver started at http://127.0.241.2:46525/ using document root <none> and password file <none>
I20260619 14:50:00.147661 4770 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:00.147794 4770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:00.148128 4770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:00.150816 4770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "02f550f830b74be3b8f403233800e002"
format_stamp: "Formatted at 2026-06-19 14:50:00 on dist-test-slave-p3ml"
server_key: "c727b6a2ae8896f19b95c7d28055e8d5"
server_key_iv: "f2bd8d7af68494253498621435a7cba0"
server_key_version: "encryptionkey@0"
I20260619 14:50:00.151652 4770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "02f550f830b74be3b8f403233800e002"
format_stamp: "Formatted at 2026-06-19 14:50:00 on dist-test-slave-p3ml"
server_key: "c727b6a2ae8896f19b95c7d28055e8d5"
server_key_iv: "f2bd8d7af68494253498621435a7cba0"
server_key_version: "encryptionkey@0"
I20260619 14:50:00.156342 4770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:50:00.159595 4785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:00.160804 4770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:50:00.160955 4770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "02f550f830b74be3b8f403233800e002"
format_stamp: "Formatted at 2026-06-19 14:50:00 on dist-test-slave-p3ml"
server_key: "c727b6a2ae8896f19b95c7d28055e8d5"
server_key_iv: "f2bd8d7af68494253498621435a7cba0"
server_key_version: "encryptionkey@0"
I20260619 14:50:00.161134 4770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:00.177556 4770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:00.181655 4770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:00.181928 4770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:00.182821 4770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:00.184274 4770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:00.184397 4770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:00.184566 4770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:00.184620 4770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:00.219420 4770 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:38245
I20260619 14:50:00.219503 4897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:38245 every 8 connection(s)
I20260619 14:50:00.220824 4770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:00.225425 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 4770
I20260619 14:50:00.225615 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:00.226089 964 external_mini_cluster.cc:1428] Setting key ed0d9c8884a2bcdbb1bfedf8aa7fc2ff
Jun 19 14:50:00 dist-test-slave-p3ml krb5kdc[4529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:00.242805 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:00.223303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:52219 (local address 127.0.241.62:40081)
0619 14:50:00.223557 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:00.223564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:00.224718 (+ 1154us) server_negotiation.cc:407] Connection header received
0619 14:50:00.226568 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:00.226575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:00.226669 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:00.226775 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:00.228841 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.230073 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.231789 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.232496 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.232645 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:00.235928 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:00.235937 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:00.235946 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:00.235990 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:00.239586 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:00.240263 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:00.240272 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:00.240278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:00.240432 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:00.240841 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:00.240849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:00.240856 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:00.241224 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:00.241371 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:00.242075 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 14:50:00.242333 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:50:00.243402 4900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:00.223823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:40081 (local address 127.0.241.2:52219)
0619 14:50:00.224556 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:00.224598 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:00.226295 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:00.227188 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:00.227199 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:00.227987 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:00.228659 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:00.228684 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.230225 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.230230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:00.231654 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:00.231666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.232689 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.232693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:00.232885 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:00.233533 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:00.233554 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:00.235704 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:00.239710 (+ 4006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:00.239717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:00.239728 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:00.240133 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:00.240536 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:00.240539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:00.240541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:00.240720 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:00.241347 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:00.241353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:00.241650 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 14:50:00.241924 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 14:50:00.242316 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":297,"threads_started":1}
I20260619 14:50:00.244846 4898 heartbeater.cc:347] Connected to a master server at 127.0.241.62:40081
I20260619 14:50:00.245170 4898 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:00.246096 4898 heartbeater.cc:511] Master 127.0.241.62:40081 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
I20260619 14:50:00.247656 4574 ts_manager.cc:194] Registered new tserver with Master: 02f550f830b74be3b8f403233800e002 (127.0.241.2:38245)
I20260619 14:50:00.248525 4574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:52219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:50:00.271788 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:40081
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:00.614104 4903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:00.614379 4903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:00.614540 4903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:00.621523 4903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:00.621635 4903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:00.621778 4903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:50:00.631292 4903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:40081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:00.632550 4903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:00.633855 4903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:00.646847 4909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:00.646809 4908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:00.646885 4911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:00.647686 4903 server_base.cc:1034] running on GCE node
I20260619 14:50:00.648324 4903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:00.649231 4903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:00.650462 4903 hybrid_clock.cc:648] HybridClock initialized: now 1781880600650396 us; error 54 us; skew 500 ppm
I20260619 14:50:00.650771 4903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:00 dist-test-slave-p3ml krb5kdc[4529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:00.653807 4903 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:50:00.655221 4903 webserver.cc:460] Webserver started at http://127.0.241.3:42725/ using document root <none> and password file <none>
I20260619 14:50:00.655937 4903 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:00.656077 4903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:00.656383 4903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:00.659672 4903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "7b3302f20a904670845273f6176f1820"
format_stamp: "Formatted at 2026-06-19 14:50:00 on dist-test-slave-p3ml"
server_key: "15add4eace04e8b1975d921756767099"
server_key_iv: "11a6588a329c90ef38abcc872462f887"
server_key_version: "encryptionkey@0"
I20260619 14:50:00.660429 4903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "7b3302f20a904670845273f6176f1820"
format_stamp: "Formatted at 2026-06-19 14:50:00 on dist-test-slave-p3ml"
server_key: "15add4eace04e8b1975d921756767099"
server_key_iv: "11a6588a329c90ef38abcc872462f887"
server_key_version: "encryptionkey@0"
I20260619 14:50:00.665716 4903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:50:00.669344 4918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:00.670622 4903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:50:00.670783 4903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "7b3302f20a904670845273f6176f1820"
format_stamp: "Formatted at 2026-06-19 14:50:00 on dist-test-slave-p3ml"
server_key: "15add4eace04e8b1975d921756767099"
server_key_iv: "11a6588a329c90ef38abcc872462f887"
server_key_version: "encryptionkey@0"
I20260619 14:50:00.670979 4903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:00.698503 4903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:00.702266 4903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:00.702543 4903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:00.703423 4903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:00.704723 4903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:00.704820 4903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:00.704950 4903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:00.705032 4903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:00.736941 4903 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:45135
I20260619 14:50:00.736986 5030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:45135 every 8 connection(s)
I20260619 14:50:00.738204 4903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:00.739400 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 4903
I20260619 14:50:00.739575 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:00.740128 964 external_mini_cluster.cc:1428] Setting key 3f87fec0e42ec29bbd77b83d7c5c5ab3
Jun 19 14:50:00 dist-test-slave-p3ml krb5kdc[4529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:00.758201 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:00.740943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:59211 (local address 127.0.241.62:40081)
0619 14:50:00.741105 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:00.741110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:00.742585 (+ 1475us) server_negotiation.cc:407] Connection header received
0619 14:50:00.744712 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:00.744721 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:00.744801 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:00.744906 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:00.746272 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.747340 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.748933 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.749347 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.749446 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:00.752637 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:00.752641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:00.752644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:00.752674 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:00.755706 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:00.756313 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:00.756316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:00.756318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:00.756419 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:00.756773 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:00.756776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:00.756778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:00.757056 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:00.757158 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:00.757812 (+ 654us) server_negotiation.cc:299] Negotiation successful
0619 14:50:00.757967 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:50:00.759066 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:00.741617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:40081 (local address 127.0.241.3:59211)
0619 14:50:00.742405 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:00.742466 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:00.744449 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:00.745092 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:00.745101 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:00.745574 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:00.746117 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:00.746134 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.747516 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.747519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:00.748810 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:00.748821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.749519 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.749524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:00.749719 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:00.750304 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:00.750324 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:00.752434 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:00.755823 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:00.755829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:00.755840 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:00.756217 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:00.756495 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:00.756498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:00.756500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:00.756683 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:00.757188 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:00.757197 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:00.757506 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 14:50:00.757726 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 14:50:00.758060 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":305,"threads_started":1}
I20260619 14:50:00.760298 5031 heartbeater.cc:347] Connected to a master server at 127.0.241.62:40081
I20260619 14:50:00.760540 5031 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:00.761142 5031 heartbeater.cc:511] Master 127.0.241.62:40081 requested a full tablet report, sending...
I20260619 14:50:00.762326 4574 ts_manager.cc:194] Registered new tserver with Master: 7b3302f20a904670845273f6176f1820 (127.0.241.3:45135)
I20260619 14:50:00.763116 4574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:59211
I20260619 14:50:00.767412 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:50:00.777391 4764 heartbeater.cc:503] Master 127.0.241.62:40081 was elected leader, sending a full tablet report...
Jun 19 14:50:00 dist-test-slave-p3ml krb5kdc[4529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:50:00 dist-test-slave-p3ml krb5kdc[4529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:50:00.813599 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:00.799677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.0.241.1:34003)
0619 14:50:00.800281 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:00.800288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:00.800314 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:50:00.800371 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:00.800377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:00.800594 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:00.800720 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:00.801945 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.802970 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.804178 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.804737 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.804832 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:00.808461 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:00.808471 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:00.808485 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:00.808522 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:00.811268 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:00.811878 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:00.811884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:00.811886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:00.811995 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:00.812297 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:00.812300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:00.812302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:00.812618 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:00.812834 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:00.813029 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 14:50:00.813210 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":326,"threads_started":1}
Jun 19 14:50:00 dist-test-slave-p3ml krb5kdc[4529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:00.829828 4766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:00.817361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.0.241.62:40081)
0619 14:50:00.817505 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:00.817509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:00.817776 (+ 267us) server_negotiation.cc:407] Connection header received
0619 14:50:00.817915 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:00.817919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:00.817978 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:00.818060 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:00.819274 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.820022 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.821128 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.821495 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.821597 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:00.825285 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:00.825290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:00.825293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:00.825327 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:00.827811 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:00.828338 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:00.828342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:00.828344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:00.828453 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:00.828724 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:00.828727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:00.828729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:00.828982 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:00.829141 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:00.829392 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 14:50:00.829586 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 14:50:00.830315 4574 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36280
I20260619 14:50:00.831254 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 4636
I20260619 14:50:00.843321 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:00.832160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.0.241.1:34003)
0619 14:50:00.832355 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:00.832361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:00.832582 (+ 221us) server_negotiation.cc:407] Connection header received
0619 14:50:00.832751 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:00.832755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:00.832823 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:00.832917 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:00.834000 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.834647 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.835877 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.836291 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.836417 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:00.838505 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:00.838510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:00.838513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:00.838541 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:00.841002 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:00.841593 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:00.841606 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:00.841609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:00.841716 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:00.842053 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:00.842059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:00.842063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:00.842289 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:00.842512 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:00.842796 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 14:50:00.843021 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:50:00.843942 4759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:00.925453 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 4636
I20260619 14:50:00.941327 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 4770
Jun 19 14:50:00 dist-test-slave-p3ml krb5kdc[4529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:50:00.956099 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:00.942092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35132 (local address 127.0.241.2:38245)
0619 14:50:00.942647 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:00.942654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:00.942679 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:50:00.942744 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:00.942750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:00.942959 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:00.943093 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:00.944284 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.945273 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.946443 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:00.946980 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:00.947090 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:00.951162 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:00.951173 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:00.951186 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:00.951213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:00.953704 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:00.954271 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:00.954277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:00.954280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:00.954381 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:00.954679 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:00.954682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:00.954684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:00.954970 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:00.955181 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:00.955391 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 14:50:00.955811 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":274,"threads_started":1}
I20260619 14:50:00.957307 4893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:01.038192 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 4770
I20260619 14:50:01.053551 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 4903
Jun 19 14:50:01 dist-test-slave-p3ml krb5kdc[4529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:50:01.068822 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:01.054474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.0.241.3:45135)
0619 14:50:01.055062 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:01.055068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:01.055097 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:50:01.055152 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:01.055159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:01.055388 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:01.055589 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:01.056725 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:01.057730 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:01.058943 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:01.059757 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:01.059851 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:01.063842 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:01.063852 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:01.063866 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:01.063896 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:01.066619 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:01.067142 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:01.067148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:01.067150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:01.067255 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:01.067560 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:01.067563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:01.067565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:01.067874 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:01.068111 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:01.068273 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 14:50:01.068445 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":308,"threads_started":1}
I20260619 14:50:01.070070 5026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:01.152371 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 4903
I20260619 14:50:01.168781 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 4542
I20260619 14:50:01.170001 4604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:01.272624 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 4542
2026-06-19T14:50:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3766 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:50:01 dist-test-slave-p3ml krb5kdc[5050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:50:01 dist-test-slave-p3ml krb5kdc[5050](info): set up 2 sockets
Jun 19 14:50:01 dist-test-slave-p3ml krb5kdc[5050](info): commencing operation
krb5kdc: starting...
W20260619 14:50:02.367879 964 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:50:02 dist-test-slave-p3ml krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:50:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:50:02.564689 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45959
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:45959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:02.901675 5063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:02.901988 5063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:02.902140 5063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:02.909413 5063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:02.909531 5063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:02.909578 5063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:02.909690 5063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:02.909752 5063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:02.919720 5063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:45959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:02.921062 5063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:02.922395 5063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:02.933918 5068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:02.933882 5069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:02.933840 5071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:02.934752 5063 server_base.cc:1034] running on GCE node
I20260619 14:50:02.935623 5063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:02.937055 5063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:02.938308 5063 hybrid_clock.cc:648] HybridClock initialized: now 1781880602938248 us; error 54 us; skew 500 ppm
I20260619 14:50:02.938761 5063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:02 dist-test-slave-p3ml krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:02.941879 5063 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:50:02.942991 5063 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:02.943080 5063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:02.943333 5063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:02.946323 5063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "ab0d0ba99d2c44a081117759ddf047d3"
format_stamp: "Formatted at 2026-06-19 14:50:02 on dist-test-slave-p3ml"
server_key: "944bc0a9c84d296d519c68d379693929"
server_key_iv: "dcfbddef056c43aecfec0c769af61a1a"
server_key_version: "encryptionkey@0"
I20260619 14:50:02.947088 5063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "ab0d0ba99d2c44a081117759ddf047d3"
format_stamp: "Formatted at 2026-06-19 14:50:02 on dist-test-slave-p3ml"
server_key: "944bc0a9c84d296d519c68d379693929"
server_key_iv: "dcfbddef056c43aecfec0c769af61a1a"
server_key_version: "encryptionkey@0"
I20260619 14:50:02.952021 5063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 14:50:02.955329 5077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:02.956638 5063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:02.956773 5063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "ab0d0ba99d2c44a081117759ddf047d3"
format_stamp: "Formatted at 2026-06-19 14:50:02 on dist-test-slave-p3ml"
server_key: "944bc0a9c84d296d519c68d379693929"
server_key_iv: "dcfbddef056c43aecfec0c769af61a1a"
server_key_version: "encryptionkey@0"
I20260619 14:50:02.956956 5063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:02.978984 5063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:02.983383 5063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:02.983768 5063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:03.003368 5128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:45959 every 8 connection(s)
I20260619 14:50:03.003396 5063 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:45959
I20260619 14:50:03.004375 5063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:03.008004 5129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:03.014256 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 5063
I20260619 14:50:03.014469 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:03.015275 964 external_mini_cluster.cc:1428] Setting key be61ea83e26703477bb642f953431303
I20260619 14:50:03.019062 5129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3: Bootstrap starting.
I20260619 14:50:03.022606 5129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:03.023756 5129 log.cc:824] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:03.026552 5129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3: No bootstrap required, opened a new log
Jun 19 14:50:03 dist-test-slave-p3ml krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:03.035320 5129 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0d0ba99d2c44a081117759ddf047d3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45959 } }
I20260619 14:50:03.035780 5129 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:03.035888 5129 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0d0ba99d2c44a081117759ddf047d3, State: Initialized, Role: FOLLOWER
I20260619 14:50:03.036525 5129 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0d0ba99d2c44a081117759ddf047d3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45959 } }
I20260619 14:50:03.036759 5129 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:03.036917 5129 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:03.037091 5129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:03.039337 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:03.017521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.0.241.62:45959)
0619 14:50:03.018284 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:03.018296 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:03.018336 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:50:03.020084 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:03.020106 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:03.020491 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:03.020823 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:03.022578 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:03.024079 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:03.025730 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:03.026375 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:03.026507 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:03.031054 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:03.031065 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:03.031079 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:03.031117 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:03.034676 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:03.035481 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:03.035495 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:03.035501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:03.035677 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:03.036259 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:03.036273 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:03.036277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:03.036950 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:03.037283 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:03.037769 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 14:50:03.038234 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":335,"threads_started":1}
I20260619 14:50:03.039655 5129 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0d0ba99d2c44a081117759ddf047d3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45959 } }
I20260619 14:50:03.040141 5129 leader_election.cc:302] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab0d0ba99d2c44a081117759ddf047d3; no voters:
I20260619 14:50:03.040875 5129 leader_election.cc:288] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:03.041007 5134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:03.042317 5134 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [term 1 LEADER]: Becoming Leader. State: Replica: ab0d0ba99d2c44a081117759ddf047d3, State: Running, Role: LEADER
I20260619 14:50:03.043083 5129 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:03.042969 5134 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0d0ba99d2c44a081117759ddf047d3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45959 } }
I20260619 14:50:03.046461 5136 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab0d0ba99d2c44a081117759ddf047d3. Latest consensus state: current_term: 1 leader_uuid: "ab0d0ba99d2c44a081117759ddf047d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0d0ba99d2c44a081117759ddf047d3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45959 } } }
I20260619 14:50:03.046749 5136 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:03.047803 5135 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab0d0ba99d2c44a081117759ddf047d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab0d0ba99d2c44a081117759ddf047d3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45959 } } }
I20260619 14:50:03.048895 5135 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:03.052080 5144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:03.057714 5144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:03.067663 5144 catalog_manager.cc:1347] Generated new cluster ID: 253ee4cee2ee4923807fa92f447714a9
I20260619 14:50:03.067787 5144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:03.100499 5144 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:03.101960 5144 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:03.111222 5144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3: Generated new TSK 0
I20260619 14:50:03.112102 5144 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:50:03.169536 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45959
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:03.499914 5155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:03.500257 5155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:03.500420 5155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:03.507659 5155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:03.507793 5155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:03.507972 5155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:50:03.517612 5155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:03.518942 5155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:03.520323 5155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:03.533169 5160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:03.533212 5161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:03.533246 5163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:03.534089 5155 server_base.cc:1034] running on GCE node
I20260619 14:50:03.534672 5155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:03.535593 5155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:03.536774 5155 hybrid_clock.cc:648] HybridClock initialized: now 1781880603536735 us; error 37 us; skew 500 ppm
I20260619 14:50:03.537066 5155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:03 dist-test-slave-p3ml krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:03.540042 5155 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:50:03.541178 5155 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:03.541298 5155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:03.541602 5155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:03.544566 5155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "0c94e11679054b2484be73a5125c8dd5"
format_stamp: "Formatted at 2026-06-19 14:50:03 on dist-test-slave-p3ml"
server_key: "ed66c60f39943a27b7dc45e9ad089756"
server_key_iv: "284be2c763387aac7e653812c98c813f"
server_key_version: "encryptionkey@0"
I20260619 14:50:03.545348 5155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "0c94e11679054b2484be73a5125c8dd5"
format_stamp: "Formatted at 2026-06-19 14:50:03 on dist-test-slave-p3ml"
server_key: "ed66c60f39943a27b7dc45e9ad089756"
server_key_iv: "284be2c763387aac7e653812c98c813f"
server_key_version: "encryptionkey@0"
I20260619 14:50:03.550374 5155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 14:50:03.553692 5169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:03.554919 5155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:50:03.555078 5155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "0c94e11679054b2484be73a5125c8dd5"
format_stamp: "Formatted at 2026-06-19 14:50:03 on dist-test-slave-p3ml"
server_key: "ed66c60f39943a27b7dc45e9ad089756"
server_key_iv: "284be2c763387aac7e653812c98c813f"
server_key_version: "encryptionkey@0"
I20260619 14:50:03.555280 5155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:03.599092 5155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:03.603121 5155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:03.603387 5155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:03.604349 5155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:03.605628 5155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:03.605755 5155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:03.605882 5155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:03.605966 5155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:03.638968 5155 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:39085
I20260619 14:50:03.639047 5281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:39085 every 8 connection(s)
I20260619 14:50:03.639813 5155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:03.650184 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 5155
I20260619 14:50:03.650441 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:03.651319 964 external_mini_cluster.cc:1428] Setting key c74cec2513be100d9df66fc38722bd7c
Jun 19 14:50:03 dist-test-slave-p3ml krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:03.660768 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:03.642222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:58779 (local address 127.0.241.62:45959)
0619 14:50:03.642668 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:03.642675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:03.643919 (+ 1244us) server_negotiation.cc:407] Connection header received
0619 14:50:03.645782 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:03.645786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:03.645849 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:03.645980 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:03.647719 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:03.648459 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:03.650217 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:03.650695 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:03.650788 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:03.654675 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:03.654679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:03.654682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:03.654710 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:03.657241 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:03.657911 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:03.657916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:03.657919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:03.658032 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:03.658611 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:03.658618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:03.658623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:03.658982 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:03.659133 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:03.660026 (+ 893us) server_negotiation.cc:299] Negotiation successful
0619 14:50:03.660266 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":207,"threads_started":1}
I20260619 14:50:03.661633 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:03.642824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45959 (local address 127.0.241.1:58779)
0619 14:50:03.643720 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:03.643769 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:03.645531 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:03.646271 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:03.646280 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:03.646829 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:03.647569 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:03.647590 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:03.648584 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:03.648587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:03.650032 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:03.650049 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:03.650929 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:03.650937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:03.651263 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:03.652034 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:03.652074 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:03.654478 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:03.657369 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:03.657378 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:03.657393 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:03.657788 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:03.658168 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:03.658177 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:03.658184 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:03.658478 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:03.659151 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:03.659159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:03.659630 (+ 471us) client_negotiation.cc:769] Sending connection context
0619 14:50:03.659883 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 14:50:03.660334 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":358,"threads_started":1}
I20260619 14:50:03.663098 5282 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45959
I20260619 14:50:03.663486 5282 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:03.664232 5282 heartbeater.cc:511] Master 127.0.241.62:45959 requested a full tablet report, sending...
I20260619 14:50:03.666379 5094 ts_manager.cc:194] Registered new tserver with Master: 0c94e11679054b2484be73a5125c8dd5 (127.0.241.1:39085)
I20260619 14:50:03.667898 5094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:58779
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:50:03.694109 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45959
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:04.003594 5288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:04.003970 5288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:04.004136 5288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:04.011255 5288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:04.011351 5288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:04.011566 5288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:50:04.020938 5288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:04.022172 5288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:04.023473 5288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:04.035902 5293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:04.035879 5296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:04.035985 5294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:04.036535 5288 server_base.cc:1034] running on GCE node
I20260619 14:50:04.037125 5288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:04.037978 5288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:04.039237 5288 hybrid_clock.cc:648] HybridClock initialized: now 1781880604039189 us; error 48 us; skew 500 ppm
I20260619 14:50:04.039644 5288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:04 dist-test-slave-p3ml krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:04.043140 5288 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:50:04.044406 5288 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:04.044535 5288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:04.044854 5288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:04.047873 5288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "04482031dc23441b83ffe7766aadaba5"
format_stamp: "Formatted at 2026-06-19 14:50:04 on dist-test-slave-p3ml"
server_key: "c74978ec1dee411c7f810bceb65ca10b"
server_key_iv: "372b8592b8da60325b550ae7c77a4470"
server_key_version: "encryptionkey@0"
I20260619 14:50:04.048650 5288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "04482031dc23441b83ffe7766aadaba5"
format_stamp: "Formatted at 2026-06-19 14:50:04 on dist-test-slave-p3ml"
server_key: "c74978ec1dee411c7f810bceb65ca10b"
server_key_iv: "372b8592b8da60325b550ae7c77a4470"
server_key_version: "encryptionkey@0"
I20260619 14:50:04.053682 5288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:50:04.056950 5302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:04.058228 5288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:50:04.058398 5288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "04482031dc23441b83ffe7766aadaba5"
format_stamp: "Formatted at 2026-06-19 14:50:04 on dist-test-slave-p3ml"
server_key: "c74978ec1dee411c7f810bceb65ca10b"
server_key_iv: "372b8592b8da60325b550ae7c77a4470"
server_key_version: "encryptionkey@0"
I20260619 14:50:04.058609 5288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:04.079860 5288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:04.084043 5288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:04.084347 5288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:04.085572 5288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:04.086913 5288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:04.087037 5288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:04.087162 5288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:04.087251 5288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:04.117695 5288 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:45699
I20260619 14:50:04.117748 5414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:45699 every 8 connection(s)
I20260619 14:50:04.118533 5288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:04.121418 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 5288
I20260619 14:50:04.121605 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:04.122188 964 external_mini_cluster.cc:1428] Setting key ed6352c637c46b3655ab21e49c768b21
Jun 19 14:50:04 dist-test-slave-p3ml krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:04.140637 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.120841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:47597 (local address 127.0.241.62:45959)
0619 14:50:04.121008 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:04.121012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:04.122224 (+ 1212us) server_negotiation.cc:407] Connection header received
0619 14:50:04.124450 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:04.124454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:04.124539 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:04.124622 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:04.126747 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.127427 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.129932 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.130514 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.130650 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.134810 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:04.134815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:04.134818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:04.134848 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:04.137419 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:04.138371 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:04.138375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:04.138377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:04.138485 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:04.138937 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:04.138940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:04.138942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:04.139189 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:04.139291 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:04.140263 (+ 972us) server_negotiation.cc:299] Negotiation successful
0619 14:50:04.140418 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:50:04.142230 5417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.121234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45959 (local address 127.0.241.2:47597)
0619 14:50:04.122034 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:04.122088 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:04.124177 (+ 2089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:04.124890 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:04.124905 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:04.125694 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:04.126532 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.126576 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.127653 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.127661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.129761 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.129778 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.130735 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.130742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.130953 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.131703 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:04.131736 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:04.134596 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:04.137547 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:04.137559 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:04.137592 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:04.138262 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:04.138589 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:04.138595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:04.138601 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:04.138832 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:04.139320 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:04.139331 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:04.139839 (+ 508us) client_negotiation.cc:769] Sending connection context
0619 14:50:04.140206 (+ 367us) client_negotiation.cc:240] Negotiation successful
0619 14:50:04.140743 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":334,"threads_started":1}
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
I20260619 14:50:04.143647 5415 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45959
I20260619 14:50:04.143939 5415 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:04.144625 5415 heartbeater.cc:511] Master 127.0.241.62:45959 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
I20260619 14:50:04.146044 5094 ts_manager.cc:194] Registered new tserver with Master: 04482031dc23441b83ffe7766aadaba5 (127.0.241.2:45699)
I20260619 14:50:04.146863 5094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:47597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:50:04.168901 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45959
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:04.491942 5420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:04.492254 5420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:04.492406 5420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:04.499725 5420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:04.499868 5420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:04.500036 5420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:50:04.509881 5420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:04.511176 5420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:04.512573 5420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:04.525388 5425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:04.525295 5428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:04.525671 5426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:04.526558 5420 server_base.cc:1034] running on GCE node
I20260619 14:50:04.527209 5420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:04.528208 5420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:04.529422 5420 hybrid_clock.cc:648] HybridClock initialized: now 1781880604529365 us; error 53 us; skew 500 ppm
I20260619 14:50:04.529750 5420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:04 dist-test-slave-p3ml krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:04.532840 5420 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:50:04.533968 5420 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:04.534121 5420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:04.534435 5420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:04.537254 5420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "07b4a7e185c1491aa5dfa46e36730768"
format_stamp: "Formatted at 2026-06-19 14:50:04 on dist-test-slave-p3ml"
server_key: "71e26e2728a3c8a1d6fd605b422e6848"
server_key_iv: "191017fc02219644d2ebffadf646f4e4"
server_key_version: "encryptionkey@0"
I20260619 14:50:04.538038 5420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "07b4a7e185c1491aa5dfa46e36730768"
format_stamp: "Formatted at 2026-06-19 14:50:04 on dist-test-slave-p3ml"
server_key: "71e26e2728a3c8a1d6fd605b422e6848"
server_key_iv: "191017fc02219644d2ebffadf646f4e4"
server_key_version: "encryptionkey@0"
I20260619 14:50:04.542824 5420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:50:04.546434 5434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:04.547828 5420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:04.547981 5420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "07b4a7e185c1491aa5dfa46e36730768"
format_stamp: "Formatted at 2026-06-19 14:50:04 on dist-test-slave-p3ml"
server_key: "71e26e2728a3c8a1d6fd605b422e6848"
server_key_iv: "191017fc02219644d2ebffadf646f4e4"
server_key_version: "encryptionkey@0"
I20260619 14:50:04.548178 5420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:04.574326 5420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:04.578387 5420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:04.578660 5420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:04.579592 5420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:04.580817 5420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:04.580932 5420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:04.581054 5420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:04.581136 5420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:04.613062 5420 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:33509
I20260619 14:50:04.613111 5546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:33509 every 8 connection(s)
I20260619 14:50:04.613906 5420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:04.617101 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 5420
I20260619 14:50:04.617322 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:04.617870 964 external_mini_cluster.cc:1428] Setting key 5bc8440d0289e28bfcd74a7168044262
Jun 19 14:50:04 dist-test-slave-p3ml krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:04.633343 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.616354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:36685 (local address 127.0.241.62:45959)
0619 14:50:04.616567 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:04.616571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:04.617899 (+ 1328us) server_negotiation.cc:407] Connection header received
0619 14:50:04.619964 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:04.619969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:04.620065 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:04.620142 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:04.621690 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.622422 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.623965 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.624343 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.624431 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.627687 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:04.627692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:04.627696 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:04.627724 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:04.630187 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:04.630867 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:04.630870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:04.630872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:04.631006 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:04.631523 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:04.631529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:04.631532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:04.631894 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:04.632050 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:04.632793 (+ 743us) server_negotiation.cc:299] Negotiation successful
0619 14:50:04.632999 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:50:04.634234 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.616887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45959 (local address 127.0.241.3:36685)
0619 14:50:04.617748 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:04.617785 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:04.619680 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:04.620417 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:04.620429 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:04.620962 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:04.621538 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.621555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.622556 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.622560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.623850 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.623861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.624555 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.624559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.624771 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.625352 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:04.625372 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:04.627439 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:04.630311 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:04.630317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:04.630328 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:04.630766 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:04.631115 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:04.631120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:04.631124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:04.631358 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:04.632040 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:04.632047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:04.632408 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 14:50:04.632675 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 14:50:04.633175 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":388,"threads_started":1}
I20260619 14:50:04.635524 5547 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45959
I20260619 14:50:04.635767 5547 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:04.636330 5547 heartbeater.cc:511] Master 127.0.241.62:45959 requested a full tablet report, sending...
I20260619 14:50:04.637591 5094 ts_manager.cc:194] Registered new tserver with Master: 07b4a7e185c1491aa5dfa46e36730768 (127.0.241.3:33509)
I20260619 14:50:04.638572 5094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:36685
I20260619 14:50:04.645115 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:50:04.661654 5284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.649447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.0.241.62:45959)
0619 14:50:04.649653 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:04.649662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:04.650035 (+ 373us) server_negotiation.cc:407] Connection header received
0619 14:50:04.650165 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:04.650169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:04.650223 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:04.650291 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:04.651775 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.652535 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.653681 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.654081 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.654181 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.656415 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:04.656420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:04.656423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:04.656451 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:04.659011 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:04.659676 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:04.659679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:04.659681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:04.659791 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:04.660187 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:04.660190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:04.660192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:04.660515 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:04.660749 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:04.661114 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 14:50:04.661281 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:50:04.670753 5282 heartbeater.cc:503] Master 127.0.241.62:45959 was elected leader, sending a full tablet report...
I20260619 14:50:04.665527 5094 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:04.672614 5094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:04.707289 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.683817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:45699 (local address 127.0.0.1:50668)
0619 14:50:04.696735 (+ 12918us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:04.696753 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:04.696873 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:04.697557 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:04.697561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:04.697588 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:04.698084 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.698099 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.700797 (+ 2698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.700805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.703889 (+ 3084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.703908 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.705945 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.705953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.706244 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.706319 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 14:50:04.706447 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 14:50:04.706693 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12771,"mutex_wait_us":53,"thread_start_us":12753,"threads_started":1}
I20260619 14:50:04.708775 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.683230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:39085 (local address 127.0.0.1:60564)
0619 14:50:04.695677 (+ 12447us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:04.695734 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:04.695966 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:04.697468 (+ 1502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:04.697476 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:04.697506 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:04.698588 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.698607 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.701841 (+ 3234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.701848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.704804 (+ 2956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.704822 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.707575 (+ 2753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.707584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.707843 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.707893 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 14:50:04.708009 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 14:50:04.708189 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12063,"mutex_wait_us":78,"thread_start_us":13228,"threads_started":1}
I20260619 14:50:04.709728 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.683599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.0.241.1:39085)
0619 14:50:04.696358 (+ 12759us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:04.696366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:04.696394 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:50:04.696497 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:04.696505 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:04.696778 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:04.696980 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:04.698819 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.701613 (+ 2794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.705050 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.706926 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.707094 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.707391 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:04.709056 (+ 1665us) server_negotiation.cc:299] Negotiation successful
0619 14:50:04.709284 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12546,"thread_start_us":12406,"threads_started":1}
I20260619 14:50:04.709818 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.684055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.0.241.2:45699)
0619 14:50:04.696545 (+ 12490us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:04.696553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:04.696992 (+ 439us) server_negotiation.cc:407] Connection header received
0619 14:50:04.697090 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:04.697100 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:04.697354 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:04.698276 (+ 922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:04.698540 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.700611 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.704140 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.705679 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.708389 (+ 2710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.709135 (+ 746us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:04.709224 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 14:50:04.709442 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12316,"thread_start_us":13105,"threads_started":1}
I20260619 14:50:04.713915 5217 tablet_service.cc:1467] Processing CreateTablet for tablet 927a33476ca4474fa753c6bf552db190 (DEFAULT_TABLE table=test-table [id=fa6ffb6c8a0d49f488fa169ae5635ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:04.713915 5350 tablet_service.cc:1467] Processing CreateTablet for tablet 927a33476ca4474fa753c6bf552db190 (DEFAULT_TABLE table=test-table [id=fa6ffb6c8a0d49f488fa169ae5635ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:04.715512 5217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927a33476ca4474fa753c6bf552db190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:04.715580 5350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927a33476ca4474fa753c6bf552db190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:04.717248 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.697907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:33509 (local address 127.0.0.1:46314)
0619 14:50:04.708048 (+ 10141us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:04.708070 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:04.708200 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:04.711478 (+ 3278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:04.711483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:04.711506 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:04.711668 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.711677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.712886 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.712894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.714567 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.714584 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.716491 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.716497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.716674 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.716706 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:50:04.716817 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:50:04.716949 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10010,"thread_start_us":12218,"threads_started":1}
I20260619 14:50:04.717315 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.697551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.0.241.3:33509)
0619 14:50:04.710927 (+ 13376us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:04.710935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:04.710967 (+ 32us) server_negotiation.cc:407] Connection header received
0619 14:50:04.711053 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:04.711061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:04.711282 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:04.711477 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:04.711805 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.712762 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.714742 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.716294 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.716483 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.716702 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:04.716849 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 14:50:04.716998 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13214,"thread_start_us":13066,"threads_started":1}
I20260619 14:50:04.720966 5482 tablet_service.cc:1467] Processing CreateTablet for tablet 927a33476ca4474fa753c6bf552db190 (DEFAULT_TABLE table=test-table [id=fa6ffb6c8a0d49f488fa169ae5635ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:04.722080 5482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927a33476ca4474fa753c6bf552db190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:04.730077 5566 tablet_bootstrap.cc:492] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5: Bootstrap starting.
I20260619 14:50:04.730866 5565 tablet_bootstrap.cc:492] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5: Bootstrap starting.
I20260619 14:50:04.733453 5567 tablet_bootstrap.cc:492] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768: Bootstrap starting.
I20260619 14:50:04.735088 5566 tablet_bootstrap.cc:654] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:04.736423 5566 log.cc:824] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:04.736439 5567 tablet_bootstrap.cc:654] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:04.736801 5565 tablet_bootstrap.cc:654] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:04.737416 5567 log.cc:824] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:04.738169 5565 log.cc:824] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:04.740373 5567 tablet_bootstrap.cc:492] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768: No bootstrap required, opened a new log
I20260619 14:50:04.740664 5567 ts_tablet_manager.cc:1397] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260619 14:50:04.740957 5566 tablet_bootstrap.cc:492] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5: No bootstrap required, opened a new log
I20260619 14:50:04.741199 5566 ts_tablet_manager.cc:1397] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260619 14:50:04.741315 5565 tablet_bootstrap.cc:492] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5: No bootstrap required, opened a new log
I20260619 14:50:04.741575 5565 ts_tablet_manager.cc:1397] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260619 14:50:04.748831 5566 raft_consensus.cc:348] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c94e11679054b2484be73a5125c8dd5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 39085 } } peers { permanent_uuid: "07b4a7e185c1491aa5dfa46e36730768" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33509 } } peers { permanent_uuid: "04482031dc23441b83ffe7766aadaba5" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 45699 } }
I20260619 14:50:04.749194 5566 raft_consensus.cc:374] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:04.749281 5566 raft_consensus.cc:729] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c94e11679054b2484be73a5125c8dd5, State: Initialized, Role: FOLLOWER
I20260619 14:50:04.749874 5566 consensus_queue.cc:260] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c94e11679054b2484be73a5125c8dd5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 39085 } } peers { permanent_uuid: "07b4a7e185c1491aa5dfa46e36730768" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33509 } } peers { permanent_uuid: "04482031dc23441b83ffe7766aadaba5" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 45699 } }
I20260619 14:50:04.751361 5567 raft_consensus.cc:348] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c94e11679054b2484be73a5125c8dd5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 39085 } } peers { permanent_uuid: "07b4a7e185c1491aa5dfa46e36730768" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33509 } } peers { permanent_uuid: "04482031dc23441b83ffe7766aadaba5" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 45699 } }
I20260619 14:50:04.751796 5567 raft_consensus.cc:374] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:04.751894 5567 raft_consensus.cc:729] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b4a7e185c1491aa5dfa46e36730768, State: Initialized, Role: FOLLOWER
I20260619 14:50:04.752065 5565 raft_consensus.cc:348] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c94e11679054b2484be73a5125c8dd5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 39085 } } peers { permanent_uuid: "07b4a7e185c1491aa5dfa46e36730768" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33509 } } peers { permanent_uuid: "04482031dc23441b83ffe7766aadaba5" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 45699 } }
I20260619 14:50:04.752449 5565 raft_consensus.cc:374] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:04.752534 5565 raft_consensus.cc:729] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04482031dc23441b83ffe7766aadaba5, State: Initialized, Role: FOLLOWER
I20260619 14:50:04.752600 5567 consensus_queue.cc:260] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c94e11679054b2484be73a5125c8dd5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 39085 } } peers { permanent_uuid: "07b4a7e185c1491aa5dfa46e36730768" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33509 } } peers { permanent_uuid: "04482031dc23441b83ffe7766aadaba5" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 45699 } }
I20260619 14:50:04.752924 5566 ts_tablet_manager.cc:1428] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260619 14:50:04.753077 5565 consensus_queue.cc:260] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c94e11679054b2484be73a5125c8dd5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 39085 } } peers { permanent_uuid: "07b4a7e185c1491aa5dfa46e36730768" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33509 } } peers { permanent_uuid: "04482031dc23441b83ffe7766aadaba5" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 45699 } }
I20260619 14:50:04.753770 5547 heartbeater.cc:503] Master 127.0.241.62:45959 was elected leader, sending a full tablet report...
I20260619 14:50:04.754587 5567 ts_tablet_manager.cc:1428] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260619 14:50:04.755612 5415 heartbeater.cc:503] Master 127.0.241.62:45959 was elected leader, sending a full tablet report...
I20260619 14:50:04.756246 5565 ts_tablet_manager.cc:1428] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260619 14:50:04.793869 5571 raft_consensus.cc:482] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:04.794207 5571 raft_consensus.cc:504] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c94e11679054b2484be73a5125c8dd5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 39085 } } peers { permanent_uuid: "07b4a7e185c1491aa5dfa46e36730768" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33509 } } peers { permanent_uuid: "04482031dc23441b83ffe7766aadaba5" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 45699 } }
I20260619 14:50:04.795997 5571 leader_election.cc:288] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07b4a7e185c1491aa5dfa46e36730768 (127.0.241.3:33509), 04482031dc23441b83ffe7766aadaba5 (127.0.241.2:45699)
I20260619 14:50:04.802384 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.796487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:45193 (local address 127.0.241.3:33509)
0619 14:50:04.796694 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:04.796701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:04.797297 (+ 596us) server_negotiation.cc:407] Connection header received
0619 14:50:04.797489 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:04.797493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:04.797546 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:04.797635 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:04.798302 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.798973 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.800702 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.801435 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.801588 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.801693 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:04.802009 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 14:50:04.802118 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:50:04.802419 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.796603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:48533 (local address 127.0.241.2:45699)
0619 14:50:04.796862 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:04.796869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:04.797220 (+ 351us) server_negotiation.cc:407] Connection header received
0619 14:50:04.797389 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:04.797393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:04.797450 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:04.797542 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:04.798182 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.798865 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.800718 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.801479 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.801642 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.801734 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:04.802037 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 14:50:04.802156 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 14:50:04.802580 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.796236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:45699 (local address 127.0.241.1:48533)
0619 14:50:04.797106 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:04.797131 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:04.797249 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:04.797581 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:04.797587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:04.797612 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:04.798027 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.798036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.798983 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.798991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.800579 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.800590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.801701 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.801707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.801890 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.801911 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:50:04.802006 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 14:50:04.802116 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":54,"spinlock_wait_cycles":17664,"thread_start_us":602,"threads_started":1}
I20260619 14:50:04.802879 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.796236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:33509 (local address 127.0.241.1:45193)
0619 14:50:04.797207 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:04.797225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:04.797357 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:04.797646 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:04.797659 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:04.797686 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:04.798162 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.798171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.799099 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.799107 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.800552 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:04.800565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.801668 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.801679 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:04.801868 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.801891 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:50:04.801988 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:50:04.802105 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":80,"thread_start_us":563,"threads_started":1}
I20260619 14:50:04.803583 5370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927a33476ca4474fa753c6bf552db190" candidate_uuid: "0c94e11679054b2484be73a5125c8dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04482031dc23441b83ffe7766aadaba5" is_pre_election: true
I20260619 14:50:04.803790 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927a33476ca4474fa753c6bf552db190" candidate_uuid: "0c94e11679054b2484be73a5125c8dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b4a7e185c1491aa5dfa46e36730768" is_pre_election: true
I20260619 14:50:04.804051 5370 raft_consensus.cc:2454] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c94e11679054b2484be73a5125c8dd5 in term 0.
I20260619 14:50:04.804260 5502 raft_consensus.cc:2454] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c94e11679054b2484be73a5125c8dd5 in term 0.
I20260619 14:50:04.804759 5171 leader_election.cc:302] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04482031dc23441b83ffe7766aadaba5, 0c94e11679054b2484be73a5125c8dd5; no voters:
I20260619 14:50:04.805142 5571 raft_consensus.cc:2781] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:04.805280 5571 raft_consensus.cc:482] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:04.805389 5571 raft_consensus.cc:3037] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:04.807423 5571 raft_consensus.cc:504] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c94e11679054b2484be73a5125c8dd5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 39085 } } peers { permanent_uuid: "07b4a7e185c1491aa5dfa46e36730768" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33509 } } peers { permanent_uuid: "04482031dc23441b83ffe7766aadaba5" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 45699 } }
I20260619 14:50:04.807992 5571 leader_election.cc:288] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [CANDIDATE]: Term 1 election: Requested vote from peers 07b4a7e185c1491aa5dfa46e36730768 (127.0.241.3:33509), 04482031dc23441b83ffe7766aadaba5 (127.0.241.2:45699)
I20260619 14:50:04.808422 5370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927a33476ca4474fa753c6bf552db190" candidate_uuid: "0c94e11679054b2484be73a5125c8dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04482031dc23441b83ffe7766aadaba5"
I20260619 14:50:04.808422 5502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927a33476ca4474fa753c6bf552db190" candidate_uuid: "0c94e11679054b2484be73a5125c8dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b4a7e185c1491aa5dfa46e36730768"
I20260619 14:50:04.808609 5502 raft_consensus.cc:3037] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:04.808607 5370 raft_consensus.cc:3037] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:04.810670 5370 raft_consensus.cc:2454] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c94e11679054b2484be73a5125c8dd5 in term 1.
I20260619 14:50:04.810674 5502 raft_consensus.cc:2454] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c94e11679054b2484be73a5125c8dd5 in term 1.
I20260619 14:50:04.811085 5171 leader_election.cc:302] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04482031dc23441b83ffe7766aadaba5, 0c94e11679054b2484be73a5125c8dd5; no voters:
I20260619 14:50:04.811383 5571 raft_consensus.cc:2781] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:04.811908 5571 raft_consensus.cc:686] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 1 LEADER]: Becoming Leader. State: Replica: 0c94e11679054b2484be73a5125c8dd5, State: Running, Role: LEADER
I20260619 14:50:04.812361 5571 consensus_queue.cc:237] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c94e11679054b2484be73a5125c8dd5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 39085 } } peers { permanent_uuid: "07b4a7e185c1491aa5dfa46e36730768" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33509 } } peers { permanent_uuid: "04482031dc23441b83ffe7766aadaba5" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 45699 } }
I20260619 14:50:04.816782 5093 catalog_manager.cc:5495] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c94e11679054b2484be73a5125c8dd5 (127.0.241.1). New cstate: current_term: 1 leader_uuid: "0c94e11679054b2484be73a5125c8dd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c94e11679054b2484be73a5125c8dd5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 39085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07b4a7e185c1491aa5dfa46e36730768" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04482031dc23441b83ffe7766aadaba5" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 45699 } health_report { overall_health: UNKNOWN } } }
I20260619 14:50:04.860736 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.855726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.0.241.1:39085)
0619 14:50:04.855944 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:04.855949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:04.855966 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:50:04.856069 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:04.856072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:04.856154 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:04.856276 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:04.856586 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.857314 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.858620 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.859037 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.859159 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.859599 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:04.860162 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:04.860272 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:04.860353 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 14:50:04.860412 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 14:50:04.865307 5548 tablet.cc:2369] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:04.869117 5502 raft_consensus.cc:1261] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768 [term 1 FOLLOWER]: Refusing update from remote peer 0c94e11679054b2484be73a5125c8dd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:04.869118 5370 raft_consensus.cc:1261] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5 [term 1 FOLLOWER]: Refusing update from remote peer 0c94e11679054b2484be73a5125c8dd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:04.870088 5571 consensus_queue.cc:1035] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04482031dc23441b83ffe7766aadaba5" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 45699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:04.870384 5576 consensus_queue.cc:1035] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b4a7e185c1491aa5dfa46e36730768" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:50:04.870613 5416 tablet.cc:2369] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:04.882335 5579 mvcc.cc:205] Tried to move back new op lower bound from 7298582957533130752 to 7298582957314895872. Current Snapshot: MvccSnapshot[applied={T|T < 7298582957533130752}]
I20260619 14:50:04.883987 5580 mvcc.cc:205] Tried to move back new op lower bound from 7298582957533130752 to 7298582957314895872. Current Snapshot: MvccSnapshot[applied={T|T < 7298582957533130752}]
I20260619 14:50:04.897738 5094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:04.898005 5094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:04.900990 5094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab0d0ba99d2c44a081117759ddf047d3: Sending DeleteTablet for 3 replicas of tablet 927a33476ca4474fa753c6bf552db190
I20260619 14:50:04.901922 5217 tablet_service.cc:1514] Processing DeleteTablet for tablet 927a33476ca4474fa753c6bf552db190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:04 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:60564
I20260619 14:50:04.902295 5350 tablet_service.cc:1514] Processing DeleteTablet for tablet 927a33476ca4474fa753c6bf552db190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:04 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:50668
I20260619 14:50:04.902388 5482 tablet_service.cc:1514] Processing DeleteTablet for tablet 927a33476ca4474fa753c6bf552db190 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:04 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:46314
I20260619 14:50:04.905190 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 5155
I20260619 14:50:04.913996 5592 tablet_replica.cc:330] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768: stopping tablet replica
I20260619 14:50:04.914774 5592 raft_consensus.cc:2229] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:04.915685 5592 raft_consensus.cc:2258] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:04.918653 5592 ts_tablet_manager.cc:1905] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 14:50:04 dist-test-slave-p3ml krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:50:04.927398 5591 tablet_replica.cc:330] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5: stopping tablet replica
I20260619 14:50:04.927959 5591 raft_consensus.cc:2229] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:04.928295 5590 tablet_replica.cc:330] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5: stopping tablet replica
I20260619 14:50:04.928640 5591 raft_consensus.cc:2258] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:04.928676 5592 ts_tablet_manager.cc:1918] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:04.928704 5590 raft_consensus.cc:2229] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:04.928835 5592 log.cc:1196] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/927a33476ca4474fa753c6bf552db190
I20260619 14:50:04.929387 5592 ts_tablet_manager.cc:1939] T 927a33476ca4474fa753c6bf552db190 P 07b4a7e185c1491aa5dfa46e36730768: Deleting consensus metadata
I20260619 14:50:04.929749 5590 raft_consensus.cc:2258] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:04.931123 5591 ts_tablet_manager.cc:1905] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:04.931262 5081 catalog_manager.cc:4841] TS 07b4a7e185c1491aa5dfa46e36730768 (127.0.241.3:33509): tablet 927a33476ca4474fa753c6bf552db190 (table test-table [id=fa6ffb6c8a0d49f488fa169ae5635ae4]) successfully deleted
I20260619 14:50:04.932799 5590 ts_tablet_manager.cc:1905] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:04.936414 5591 ts_tablet_manager.cc:1918] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:04.936573 5591 log.cc:1196] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/927a33476ca4474fa753c6bf552db190
I20260619 14:50:04.937132 5591 ts_tablet_manager.cc:1939] T 927a33476ca4474fa753c6bf552db190 P 04482031dc23441b83ffe7766aadaba5: Deleting consensus metadata
I20260619 14:50:04.938962 5079 catalog_manager.cc:4841] TS 04482031dc23441b83ffe7766aadaba5 (127.0.241.2:45699): tablet 927a33476ca4474fa753c6bf552db190 (table test-table [id=fa6ffb6c8a0d49f488fa169ae5635ae4]) successfully deleted
I20260619 14:50:04.940235 5590 ts_tablet_manager.cc:1918] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:04.940394 5590 log.cc:1196] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/927a33476ca4474fa753c6bf552db190
I20260619 14:50:04.940968 5590 ts_tablet_manager.cc:1939] T 927a33476ca4474fa753c6bf552db190 P 0c94e11679054b2484be73a5125c8dd5: Deleting consensus metadata
I20260619 14:50:04.942669 5081 catalog_manager.cc:4841] TS 0c94e11679054b2484be73a5125c8dd5 (127.0.241.1:39085): tablet 927a33476ca4474fa753c6bf552db190 (table test-table [id=fa6ffb6c8a0d49f488fa169ae5635ae4]) successfully deleted
I20260619 14:50:04.943027 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:04.908251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.0.241.1:39085)
0619 14:50:04.908480 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:04.908490 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:04.912170 (+ 3680us) server_negotiation.cc:407] Connection header received
0619 14:50:04.912958 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:04.912964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:04.913046 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:04.913121 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:04.916420 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.917769 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.921480 (+ 3711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:04.922211 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:04.922306 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:04.935686 (+ 13380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:04.935700 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:04.935719 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:04.935782 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:04.939953 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:04.940807 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:04.940814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:04.940817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:04.940969 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:04.941386 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:04.941391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:04.941394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:04.941758 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:04.942091 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:04.942385 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 14:50:04.942622 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 14:50:04.943727 5277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:05.045627 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 5155
I20260619 14:50:05.063552 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 5288
Jun 19 14:50:05 dist-test-slave-p3ml krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:50:05.078140 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:05.064366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50684 (local address 127.0.241.2:45699)
0619 14:50:05.064572 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:05.064579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:05.064687 (+ 108us) server_negotiation.cc:407] Connection header received
0619 14:50:05.064899 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:05.064903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:05.064974 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:05.065072 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:05.066336 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:05.067027 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:05.068392 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:05.068816 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:05.068946 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:05.072829 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:05.072840 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:05.072854 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:05.072887 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:05.075382 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:05.076143 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:05.076152 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:05.076154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:05.076266 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:05.076653 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:05.076660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:05.076663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:05.077013 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:05.077304 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:05.077584 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 14:50:05.077796 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:50:05.078773 5410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:05.166389 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 5288
I20260619 14:50:05.182380 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 5420
Jun 19 14:50:05 dist-test-slave-p3ml krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:50:05.196403 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:05.183136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.0.241.3:33509)
0619 14:50:05.183292 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:05.183297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:05.183314 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:50:05.183535 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:05.183539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:05.183603 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:05.183690 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:05.184903 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:05.185696 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:05.186845 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:05.187229 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:05.187322 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:05.191431 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:05.191514 (+ 83us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:05.191536 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:05.191586 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:05.194125 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:05.194683 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:05.194690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:05.194692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:05.194806 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:05.195094 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:05.195097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:05.195099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:05.195416 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:05.195756 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:05.195988 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 14:50:05.196150 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:50:05.197037 5542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:05.285360 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 5420
I20260619 14:50:05.301388 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 5063
I20260619 14:50:05.302502 5124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:05.406038 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 5063
2026-06-19T14:50:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4134 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T14:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:05Z Disabled control of system clock
I20260619 14:50:05.472170 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:39475
--webserver_interface=127.0.241.62
--webserver_port=0
--builtin_ntp_servers=127.0.241.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:39475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:50:05.790910 5601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:05.791175 5601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:05.791291 5601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:05.798411 5601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:05.798523 5601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:05.798559 5601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:05.798606 5601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:05.798674 5601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:05.807583 5601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:39475
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:39475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:05.808827 5601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:05.810504 5601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:05.822355 5606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:05.822394 5609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:05.822454 5607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:05.823535 5601 server_base.cc:1034] running on GCE node
I20260619 14:50:05.824331 5601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:05.825709 5601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:05.826925 5601 hybrid_clock.cc:648] HybridClock initialized: now 1781880605826887 us; error 35 us; skew 500 ppm
I20260619 14:50:05.827266 5601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:05.829488 5601 webserver.cc:460] Webserver started at http://127.0.241.62:45111/ using document root <none> and password file <none>
I20260619 14:50:05.830153 5601 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:05.830286 5601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:05.830596 5601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:05.833407 5601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "0a411d392081485c9919a23c19d35292"
format_stamp: "Formatted at 2026-06-19 14:50:05 on dist-test-slave-p3ml"
server_key: "df05f3c0bcedd3b00bc8dcf9cf3b8cb9"
server_key_iv: "abc48f0ca4bd91282e9c276ba5ccf21b"
server_key_version: "encryptionkey@0"
I20260619 14:50:05.834172 5601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "0a411d392081485c9919a23c19d35292"
format_stamp: "Formatted at 2026-06-19 14:50:05 on dist-test-slave-p3ml"
server_key: "df05f3c0bcedd3b00bc8dcf9cf3b8cb9"
server_key_iv: "abc48f0ca4bd91282e9c276ba5ccf21b"
server_key_version: "encryptionkey@0"
I20260619 14:50:05.838995 5601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 14:50:05.842257 5615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:05.843636 5601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 14:50:05.843803 5601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "0a411d392081485c9919a23c19d35292"
format_stamp: "Formatted at 2026-06-19 14:50:05 on dist-test-slave-p3ml"
server_key: "df05f3c0bcedd3b00bc8dcf9cf3b8cb9"
server_key_iv: "abc48f0ca4bd91282e9c276ba5ccf21b"
server_key_version: "encryptionkey@0"
I20260619 14:50:05.844002 5601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:05.849416 5601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:05.850119 5601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:05.850412 5601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:05.867821 5601 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:39475
I20260619 14:50:05.867908 5666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:39475 every 8 connection(s)
I20260619 14:50:05.869093 5601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:05.872605 5667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:05.879326 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 5601
I20260619 14:50:05.879562 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:05.880311 964 external_mini_cluster.cc:1428] Setting key f52fd9ea96c7f99a21e2f6d3e511a693
I20260619 14:50:05.883526 5667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292: Bootstrap starting.
I20260619 14:50:05.886727 5667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:05.887903 5667 log.cc:824] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:05.888458 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:05.882320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55894 (local address 127.0.241.62:39475)
0619 14:50:05.883203 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:05.883213 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:05.883252 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:50:05.884993 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:05.885016 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:05.885402 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:05.885759 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:05.886758 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:05.886764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:05.886794 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:05.886811 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:05.886888 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:05.886904 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:05.887104 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:05.887239 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 14:50:05.887532 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":297,"threads_started":1}
I20260619 14:50:05.890514 5667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292: No bootstrap required, opened a new log
I20260619 14:50:05.897846 5667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a411d392081485c9919a23c19d35292" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39475 } }
I20260619 14:50:05.898186 5667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:05.898293 5667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a411d392081485c9919a23c19d35292, State: Initialized, Role: FOLLOWER
I20260619 14:50:05.898875 5667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a411d392081485c9919a23c19d35292" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39475 } }
I20260619 14:50:05.899102 5667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:05.899228 5667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:05.899394 5667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:05.901544 5667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a411d392081485c9919a23c19d35292" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39475 } }
I20260619 14:50:05.901999 5667 leader_election.cc:302] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a411d392081485c9919a23c19d35292; no voters:
I20260619 14:50:05.902536 5667 leader_election.cc:288] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:05.902607 5672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:05.903846 5672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [term 1 LEADER]: Becoming Leader. State: Replica: 0a411d392081485c9919a23c19d35292, State: Running, Role: LEADER
I20260619 14:50:05.904358 5672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a411d392081485c9919a23c19d35292" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39475 } }
I20260619 14:50:05.904618 5667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:05.907019 5674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a411d392081485c9919a23c19d35292. Latest consensus state: current_term: 1 leader_uuid: "0a411d392081485c9919a23c19d35292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a411d392081485c9919a23c19d35292" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39475 } } }
I20260619 14:50:05.907058 5673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a411d392081485c9919a23c19d35292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a411d392081485c9919a23c19d35292" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39475 } } }
I20260619 14:50:05.907397 5674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:05.907477 5673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:05.908543 5680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:05.913699 5680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:05.922361 5680 catalog_manager.cc:1347] Generated new cluster ID: cfa8a0c6d37f4148ae2d4cf4b86f2e5c
I20260619 14:50:05.922470 5680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:05.938055 5680 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:05.938526 5680 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:05.945581 5680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292: Generated new TSK 0
I20260619 14:50:05.946416 5680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:50:05.959968 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:39475
--builtin_ntp_servers=127.0.241.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:50:06.297230 5691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:06.297515 5691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:06.297605 5691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:06.305006 5691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:06.305115 5691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:06.305254 5691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:50:06.314390 5691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:39475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:06.315608 5691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:06.317296 5691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:06.330220 5696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:06.330243 5697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:06.330272 5699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:06.331110 5691 server_base.cc:1034] running on GCE node
I20260619 14:50:06.331744 5691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:06.332615 5691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:06.333811 5691 hybrid_clock.cc:648] HybridClock initialized: now 1781880606333763 us; error 46 us; skew 500 ppm
I20260619 14:50:06.334093 5691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:06.336181 5691 webserver.cc:460] Webserver started at http://127.0.241.1:33327/ using document root <none> and password file <none>
I20260619 14:50:06.336810 5691 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:06.336899 5691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:06.337155 5691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:06.339996 5691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "cce980fa59684e77ac8110ab15ee5cc8"
format_stamp: "Formatted at 2026-06-19 14:50:06 on dist-test-slave-p3ml"
server_key: "e0d69a790c884eacd611b38b220ac05f"
server_key_iv: "5b50cf5b7ff6518993eb41ff70166416"
server_key_version: "encryptionkey@0"
I20260619 14:50:06.340710 5691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "cce980fa59684e77ac8110ab15ee5cc8"
format_stamp: "Formatted at 2026-06-19 14:50:06 on dist-test-slave-p3ml"
server_key: "e0d69a790c884eacd611b38b220ac05f"
server_key_iv: "5b50cf5b7ff6518993eb41ff70166416"
server_key_version: "encryptionkey@0"
I20260619 14:50:06.345671 5691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:50:06.349052 5705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:06.350380 5691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:50:06.350505 5691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "cce980fa59684e77ac8110ab15ee5cc8"
format_stamp: "Formatted at 2026-06-19 14:50:06 on dist-test-slave-p3ml"
server_key: "e0d69a790c884eacd611b38b220ac05f"
server_key_iv: "5b50cf5b7ff6518993eb41ff70166416"
server_key_version: "encryptionkey@0"
I20260619 14:50:06.350701 5691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:06.356254 5691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:06.356945 5691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:06.357172 5691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:06.357972 5691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:06.359313 5691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:06.359395 5691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:06.359565 5691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:06.359627 5691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:06.391659 5691 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:45463
I20260619 14:50:06.391676 5817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:45463 every 8 connection(s)
I20260619 14:50:06.393005 5691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:06.400895 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:06.395494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:47719 (local address 127.0.241.62:39475)
0619 14:50:06.395921 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:06.395925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:06.397006 (+ 1081us) server_negotiation.cc:407] Connection header received
0619 14:50:06.398669 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:06.398674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:06.398739 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:06.398861 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:06.400162 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:06.400168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:06.400172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:06.400190 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:06.400233 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:06.400244 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:06.400373 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:06.400559 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 14:50:06.400623 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":232,"threads_started":1}
I20260619 14:50:06.401813 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:06.395973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39475 (local address 127.0.241.1:47719)
0619 14:50:06.396845 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:06.396885 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:06.398411 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:06.399116 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:06.399124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:06.399213 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:06.399871 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:06.399873 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:06.399908 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:06.399988 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:06.400350 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:06.400361 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:06.400368 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:50:06.400636 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 14:50:06.400876 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":252,"threads_started":1}
I20260619 14:50:06.402930 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 5691
I20260619 14:50:06.403136 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:06.403223 5818 heartbeater.cc:347] Connected to a master server at 127.0.241.62:39475
I20260619 14:50:06.403549 5818 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:06.403959 964 external_mini_cluster.cc:1428] Setting key cafcb05326a26486fc3b99a10820ea75
I20260619 14:50:06.404232 5818 heartbeater.cc:511] Master 127.0.241.62:39475 requested a full tablet report, sending...
I20260619 14:50:06.406384 5631 ts_manager.cc:194] Registered new tserver with Master: cce980fa59684e77ac8110ab15ee5cc8 (127.0.241.1:45463)
I20260619 14:50:06.406824 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:39475
--builtin_ntp_servers=127.0.241.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:50:06.718501 5822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:06.718783 5822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:06.718930 5822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:06.726214 5822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:06.726310 5822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:06.726492 5822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:50:06.735529 5822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:39475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:06.736827 5822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:06.738549 5822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:06.751188 5827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:06.751262 5828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:06.751286 5830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:06.752122 5822 server_base.cc:1034] running on GCE node
I20260619 14:50:06.752725 5822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:06.753700 5822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:06.754942 5822 hybrid_clock.cc:648] HybridClock initialized: now 1781880606754870 us; error 54 us; skew 500 ppm
I20260619 14:50:06.755316 5822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:06.757507 5822 webserver.cc:460] Webserver started at http://127.0.241.2:41031/ using document root <none> and password file <none>
I20260619 14:50:06.758162 5822 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:06.758296 5822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:06.758661 5822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:06.761425 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "ac37b172aa254984b2a8fff585393998"
format_stamp: "Formatted at 2026-06-19 14:50:06 on dist-test-slave-p3ml"
server_key: "7fb8d52a064a426076951b5720d48048"
server_key_iv: "7e3341957f8d92a39fba040024726544"
server_key_version: "encryptionkey@0"
I20260619 14:50:06.762194 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "ac37b172aa254984b2a8fff585393998"
format_stamp: "Formatted at 2026-06-19 14:50:06 on dist-test-slave-p3ml"
server_key: "7fb8d52a064a426076951b5720d48048"
server_key_iv: "7e3341957f8d92a39fba040024726544"
server_key_version: "encryptionkey@0"
I20260619 14:50:06.767136 5822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 14:50:06.770500 5836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:06.771848 5822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 14:50:06.772028 5822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "ac37b172aa254984b2a8fff585393998"
format_stamp: "Formatted at 2026-06-19 14:50:06 on dist-test-slave-p3ml"
server_key: "7fb8d52a064a426076951b5720d48048"
server_key_iv: "7e3341957f8d92a39fba040024726544"
server_key_version: "encryptionkey@0"
I20260619 14:50:06.772248 5822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:06.777599 5822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:06.778371 5822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:06.778642 5822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:06.779585 5822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:06.780956 5822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:06.781077 5822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:06.781198 5822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:06.781280 5822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:06.812104 5822 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:41863
I20260619 14:50:06.812191 5948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:41863 every 8 connection(s)
I20260619 14:50:06.813386 5822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:06.814601 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 5822
I20260619 14:50:06.814806 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:06.815260 964 external_mini_cluster.cc:1428] Setting key 5592ff002c60684a5cbf317d0afeaa62
I20260619 14:50:06.818655 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:39475
--builtin_ntp_servers=127.0.241.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 14:50:06.822984 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:06.817155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:36755 (local address 127.0.241.62:39475)
0619 14:50:06.817365 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:06.817372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:06.818916 (+ 1544us) server_negotiation.cc:407] Connection header received
0619 14:50:06.820696 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:06.820705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:06.820777 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:06.820927 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:06.822145 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:06.822154 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:06.822162 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:06.822186 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:06.822230 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:06.822248 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:06.822388 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:06.822520 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 14:50:06.822587 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:50:06.823724 5951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:06.817743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39475 (local address 127.0.241.2:36755)
0619 14:50:06.818777 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:06.818820 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:06.820417 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:06.821085 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:06.821092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:06.821188 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:06.821854 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:06.821856 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:06.821893 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:06.821972 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:06.822343 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:06.822349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:06.822354 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:50:06.822548 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 14:50:06.822752 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":321,"threads_started":1}
I20260619 14:50:06.825085 5949 heartbeater.cc:347] Connected to a master server at 127.0.241.62:39475
I20260619 14:50:06.825414 5949 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:06.826104 5949 heartbeater.cc:511] Master 127.0.241.62:39475 requested a full tablet report, sending...
I20260619 14:50:06.827592 5631 ts_manager.cc:194] Registered new tserver with Master: ac37b172aa254984b2a8fff585393998 (127.0.241.2:41863)
W20260619 14:50:07.152740 5952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:07.152989 5952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:07.153151 5952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:07.160538 5952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:07.160630 5952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:07.160791 5952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:50:07.169682 5952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:39475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:07.171012 5952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:07.172727 5952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:07.185547 5957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:07.185920 5958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:07.185968 5960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:07.186504 5952 server_base.cc:1034] running on GCE node
I20260619 14:50:07.187032 5952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:07.187940 5952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:07.189167 5952 hybrid_clock.cc:648] HybridClock initialized: now 1781880607189128 us; error 57 us; skew 500 ppm
I20260619 14:50:07.189455 5952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:07.191686 5952 webserver.cc:460] Webserver started at http://127.0.241.3:40051/ using document root <none> and password file <none>
I20260619 14:50:07.192309 5952 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:07.192428 5952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:07.192727 5952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:07.195374 5952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "af1c76d650104af4b0f6940b31866e32"
format_stamp: "Formatted at 2026-06-19 14:50:07 on dist-test-slave-p3ml"
server_key: "2c3c873026617097a815f1e6d9ced6b6"
server_key_iv: "cf6f610bceb94c789976618611c997e9"
server_key_version: "encryptionkey@0"
I20260619 14:50:07.196221 5952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "af1c76d650104af4b0f6940b31866e32"
format_stamp: "Formatted at 2026-06-19 14:50:07 on dist-test-slave-p3ml"
server_key: "2c3c873026617097a815f1e6d9ced6b6"
server_key_iv: "cf6f610bceb94c789976618611c997e9"
server_key_version: "encryptionkey@0"
I20260619 14:50:07.201002 5952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:50:07.204129 5966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:07.205466 5952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:07.205631 5952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "af1c76d650104af4b0f6940b31866e32"
format_stamp: "Formatted at 2026-06-19 14:50:07 on dist-test-slave-p3ml"
server_key: "2c3c873026617097a815f1e6d9ced6b6"
server_key_iv: "cf6f610bceb94c789976618611c997e9"
server_key_version: "encryptionkey@0"
I20260619 14:50:07.205842 5952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:07.211473 5952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:07.212199 5952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:07.212450 5952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:07.213256 5952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:07.214548 5952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:07.214675 5952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:07.214787 5952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:07.214880 5952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:07.247936 5952 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:38893
I20260619 14:50:07.247983 6078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:38893 every 8 connection(s)
I20260619 14:50:07.249323 5952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:07.250735 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 5952
I20260619 14:50:07.250912 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:07.251525 964 external_mini_cluster.cc:1428] Setting key 0616ad1a0c4b5abd823fdbccf3e4fc9c
I20260619 14:50:07.257416 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.251881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:59421 (local address 127.0.241.62:39475)
0619 14:50:07.252067 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:07.252073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:07.253766 (+ 1693us) server_negotiation.cc:407] Connection header received
0619 14:50:07.255667 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:07.255671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:07.255723 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:07.255795 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:07.256900 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:07.256904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:07.256908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:07.256921 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:07.256950 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:07.256960 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:07.257044 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:07.257174 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 14:50:07.257220 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:50:07.258289 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.252509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39475 (local address 127.0.241.3:59421)
0619 14:50:07.253606 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.253656 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:07.255363 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:07.255991 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:07.255998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:07.256088 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:07.256650 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:07.256652 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:07.256686 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:07.256764 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:07.257023 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:07.257028 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:07.257032 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:50:07.257222 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 14:50:07.257406 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":339,"threads_started":1}
I20260619 14:50:07.259478 6079 heartbeater.cc:347] Connected to a master server at 127.0.241.62:39475
I20260619 14:50:07.259773 6079 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:07.260335 6079 heartbeater.cc:511] Master 127.0.241.62:39475 requested a full tablet report, sending...
I20260619 14:50:07.261659 5631 ts_manager.cc:194] Registered new tserver with Master: af1c76d650104af4b0f6940b31866e32 (127.0.241.3:38893)
I20260619 14:50:07.266429 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:50:07.272933 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.270595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55910 (local address 127.0.241.62:39475)
0619 14:50:07.270748 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:07.270756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:07.271319 (+ 563us) server_negotiation.cc:407] Connection header received
0619 14:50:07.271480 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:07.271484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:07.271534 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:07.271627 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:07.272419 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:07.272423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:07.272426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:07.272440 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:07.272467 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:07.272478 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:07.272566 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:07.272613 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 14:50:07.272652 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:50:07.276445 5631 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:07.283870 5631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:07.310204 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.295003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:38893 (local address 127.0.0.1:57780)
0619 14:50:07.301960 (+ 6957us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.302019 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:07.302194 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:07.307235 (+ 5041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:07.307247 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:07.307376 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:07.308651 (+ 1275us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:07.308654 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:07.308682 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:07.308764 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:07.309371 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:07.309385 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:07.309392 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:50:07.309588 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 14:50:07.309683 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6608,"thread_start_us":360,"threads_started":1}
I20260619 14:50:07.310881 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.295180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.0.241.3:38893)
0619 14:50:07.306474 (+ 11294us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:07.306485 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:07.306521 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:50:07.306593 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:07.306600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:07.306877 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:07.307042 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:07.308932 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:07.308941 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:07.308961 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:07.309003 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:07.309122 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:07.309149 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:07.309314 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:07.309968 (+ 654us) server_negotiation.cc:299] Negotiation successful
0619 14:50:07.310091 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11112,"thread_start_us":10961,"threads_started":1}
I20260619 14:50:07.310891 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.295801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:41863 (local address 127.0.0.1:48514)
0619 14:50:07.306098 (+ 10297us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.306120 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:07.306229 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:07.308188 (+ 1959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:07.308193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:07.308217 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:07.309092 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:07.309095 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:07.309112 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:07.309161 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:07.310194 (+ 1033us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:07.310201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:07.310206 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:50:07.310335 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 14:50:07.310416 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10130,"thread_start_us":9946,"threads_started":1}
I20260619 14:50:07.311311 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.296043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48514 (local address 127.0.241.2:41863)
0619 14:50:07.307162 (+ 11119us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:07.307170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:07.307196 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:50:07.307257 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:07.307265 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:07.307489 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:07.307656 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:07.309770 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:07.309778 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:07.309793 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:07.309818 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:07.309868 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:07.309886 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:07.310724 (+ 838us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:07.310839 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 14:50:07.310940 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10952,"thread_start_us":10818,"threads_started":1}
I20260619 14:50:07.311743 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.296199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45463 (local address 127.0.0.1:46070)
0619 14:50:07.308301 (+ 12102us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.308319 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:07.308420 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:07.309304 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:07.309308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:07.309328 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:07.310539 (+ 1211us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:07.310543 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:07.310559 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:07.310611 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:07.311089 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:07.311093 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:07.311098 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:50:07.311237 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 14:50:07.311288 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11985,"thread_start_us":11850,"threads_started":1}
I20260619 14:50:07.312387 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.296640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.0.241.1:45463)
0619 14:50:07.306730 (+ 10090us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:07.306746 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:07.308515 (+ 1769us) server_negotiation.cc:407] Connection header received
0619 14:50:07.308606 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:07.308874 (+ 268us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:07.309077 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:07.309220 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:07.310784 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:07.310793 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:07.310804 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:07.310827 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:07.310884 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:07.310905 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:07.311040 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:07.311948 (+ 908us) server_negotiation.cc:299] Negotiation successful
0619 14:50:07.312082 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9950,"thread_start_us":9822,"threads_started":1}
I20260619 14:50:07.314746 5884 tablet_service.cc:1467] Processing CreateTablet for tablet 2b286ffd08e542f8b1b808dba5fdf0f8 (DEFAULT_TABLE table=test-table [id=54ba47f9e69d4965bb3bd366e5560c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:07.315044 6014 tablet_service.cc:1467] Processing CreateTablet for tablet 2b286ffd08e542f8b1b808dba5fdf0f8 (DEFAULT_TABLE table=test-table [id=54ba47f9e69d4965bb3bd366e5560c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:07.315543 5753 tablet_service.cc:1467] Processing CreateTablet for tablet 2b286ffd08e542f8b1b808dba5fdf0f8 (DEFAULT_TABLE table=test-table [id=54ba47f9e69d4965bb3bd366e5560c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:07.316195 5884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b286ffd08e542f8b1b808dba5fdf0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:07.316411 6014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b286ffd08e542f8b1b808dba5fdf0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:07.316792 5753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b286ffd08e542f8b1b808dba5fdf0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:07.327232 6097 tablet_bootstrap.cc:492] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998: Bootstrap starting.
I20260619 14:50:07.330227 6097 tablet_bootstrap.cc:654] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:07.331549 6098 tablet_bootstrap.cc:492] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8: Bootstrap starting.
I20260619 14:50:07.331600 6097 log.cc:824] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:07.333175 6099 tablet_bootstrap.cc:492] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32: Bootstrap starting.
I20260619 14:50:07.335250 6098 tablet_bootstrap.cc:654] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:07.336110 6099 tablet_bootstrap.cc:654] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:07.336439 6098 log.cc:824] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:07.336510 6097 tablet_bootstrap.cc:492] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998: No bootstrap required, opened a new log
I20260619 14:50:07.336732 6097 ts_tablet_manager.cc:1397] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 14:50:07.337127 6099 log.cc:824] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:07.341974 6098 tablet_bootstrap.cc:492] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8: No bootstrap required, opened a new log
I20260619 14:50:07.341996 6099 tablet_bootstrap.cc:492] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32: No bootstrap required, opened a new log
I20260619 14:50:07.342234 6098 ts_tablet_manager.cc:1397] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260619 14:50:07.342234 6099 ts_tablet_manager.cc:1397] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 14:50:07.345206 6097 raft_consensus.cc:348] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1c76d650104af4b0f6940b31866e32" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38893 } } peers { permanent_uuid: "cce980fa59684e77ac8110ab15ee5cc8" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45463 } } peers { permanent_uuid: "ac37b172aa254984b2a8fff585393998" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41863 } }
I20260619 14:50:07.345549 6097 raft_consensus.cc:374] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:07.345671 6097 raft_consensus.cc:729] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac37b172aa254984b2a8fff585393998, State: Initialized, Role: FOLLOWER
I20260619 14:50:07.346343 6097 consensus_queue.cc:260] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1c76d650104af4b0f6940b31866e32" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38893 } } peers { permanent_uuid: "cce980fa59684e77ac8110ab15ee5cc8" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45463 } } peers { permanent_uuid: "ac37b172aa254984b2a8fff585393998" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41863 } }
I20260619 14:50:07.347519 5949 heartbeater.cc:503] Master 127.0.241.62:39475 was elected leader, sending a full tablet report...
I20260619 14:50:07.348155 6097 ts_tablet_manager.cc:1428] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260619 14:50:07.351168 6098 raft_consensus.cc:348] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1c76d650104af4b0f6940b31866e32" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38893 } } peers { permanent_uuid: "cce980fa59684e77ac8110ab15ee5cc8" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45463 } } peers { permanent_uuid: "ac37b172aa254984b2a8fff585393998" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41863 } }
I20260619 14:50:07.351691 6098 raft_consensus.cc:374] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:07.351786 6098 raft_consensus.cc:729] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce980fa59684e77ac8110ab15ee5cc8, State: Initialized, Role: FOLLOWER
I20260619 14:50:07.352372 6098 consensus_queue.cc:260] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1c76d650104af4b0f6940b31866e32" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38893 } } peers { permanent_uuid: "cce980fa59684e77ac8110ab15ee5cc8" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45463 } } peers { permanent_uuid: "ac37b172aa254984b2a8fff585393998" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41863 } }
I20260619 14:50:07.353524 5818 heartbeater.cc:503] Master 127.0.241.62:39475 was elected leader, sending a full tablet report...
I20260619 14:50:07.353334 6099 raft_consensus.cc:348] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1c76d650104af4b0f6940b31866e32" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38893 } } peers { permanent_uuid: "cce980fa59684e77ac8110ab15ee5cc8" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45463 } } peers { permanent_uuid: "ac37b172aa254984b2a8fff585393998" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41863 } }
I20260619 14:50:07.353747 6099 raft_consensus.cc:374] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:07.353828 6099 raft_consensus.cc:729] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1c76d650104af4b0f6940b31866e32, State: Initialized, Role: FOLLOWER
I20260619 14:50:07.354074 6098 ts_tablet_manager.cc:1428] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260619 14:50:07.354394 6099 consensus_queue.cc:260] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1c76d650104af4b0f6940b31866e32" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38893 } } peers { permanent_uuid: "cce980fa59684e77ac8110ab15ee5cc8" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45463 } } peers { permanent_uuid: "ac37b172aa254984b2a8fff585393998" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41863 } }
I20260619 14:50:07.355489 6079 heartbeater.cc:503] Master 127.0.241.62:39475 was elected leader, sending a full tablet report...
I20260619 14:50:07.356097 6099 ts_tablet_manager.cc:1428] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 14:50:07.371752 6104 raft_consensus.cc:482] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:07.372010 6104 raft_consensus.cc:504] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1c76d650104af4b0f6940b31866e32" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38893 } } peers { permanent_uuid: "cce980fa59684e77ac8110ab15ee5cc8" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45463 } } peers { permanent_uuid: "ac37b172aa254984b2a8fff585393998" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41863 } }
I20260619 14:50:07.373590 6104 leader_election.cc:288] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af1c76d650104af4b0f6940b31866e32 (127.0.241.3:38893), ac37b172aa254984b2a8fff585393998 (127.0.241.2:41863)
I20260619 14:50:07.376809 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.373947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:44657 (local address 127.0.241.3:38893)
0619 14:50:07.374124 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:07.374130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:07.374802 (+ 672us) server_negotiation.cc:407] Connection header received
0619 14:50:07.374971 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:07.374976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:07.375032 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:07.375131 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:07.376087 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:07.376092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:07.376096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:07.376113 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:07.376147 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:07.376158 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:07.376272 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:07.376462 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 14:50:07.376522 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:50:07.376982 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.374002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:36391 (local address 127.0.241.2:41863)
0619 14:50:07.374152 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:07.374160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:07.374806 (+ 646us) server_negotiation.cc:407] Connection header received
0619 14:50:07.375016 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:07.375020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:07.375061 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:07.375154 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:07.376404 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:07.376409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:07.376414 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:07.376437 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:07.376463 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:07.376474 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:07.376591 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:07.376680 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 14:50:07.376737 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:50:07.377244 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.373690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:38893 (local address 127.0.241.1:44657)
0619 14:50:07.374685 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.374708 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:07.374845 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:07.375193 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:07.375197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:07.375222 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:07.375909 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:07.375912 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:07.375925 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:07.375958 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:07.376350 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:07.376353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:07.376357 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:50:07.376538 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 14:50:07.376586 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":567,"threads_started":1}
I20260619 14:50:07.377501 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.373931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:41863 (local address 127.0.241.1:36391)
0619 14:50:07.374709 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.374727 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:07.374888 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:07.375212 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:07.375217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:07.375237 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:07.376103 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:07.376105 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:07.376117 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:07.376148 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:07.376560 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:07.376565 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:07.376570 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:50:07.376730 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 14:50:07.376775 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"spinlock_wait_cycles":42752,"thread_start_us":444,"threads_started":1}
I20260619 14:50:07.378175 6034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b286ffd08e542f8b1b808dba5fdf0f8" candidate_uuid: "cce980fa59684e77ac8110ab15ee5cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af1c76d650104af4b0f6940b31866e32" is_pre_election: true
I20260619 14:50:07.378386 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b286ffd08e542f8b1b808dba5fdf0f8" candidate_uuid: "cce980fa59684e77ac8110ab15ee5cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac37b172aa254984b2a8fff585393998" is_pre_election: true
I20260619 14:50:07.378633 6034 raft_consensus.cc:2454] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cce980fa59684e77ac8110ab15ee5cc8 in term 0.
I20260619 14:50:07.378824 5904 raft_consensus.cc:2454] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cce980fa59684e77ac8110ab15ee5cc8 in term 0.
I20260619 14:50:07.379328 5706 leader_election.cc:302] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af1c76d650104af4b0f6940b31866e32, cce980fa59684e77ac8110ab15ee5cc8; no voters:
I20260619 14:50:07.379741 6104 raft_consensus.cc:2781] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:07.379879 6104 raft_consensus.cc:482] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:07.379974 6104 raft_consensus.cc:3037] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:07.382040 6104 raft_consensus.cc:504] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1c76d650104af4b0f6940b31866e32" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38893 } } peers { permanent_uuid: "cce980fa59684e77ac8110ab15ee5cc8" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45463 } } peers { permanent_uuid: "ac37b172aa254984b2a8fff585393998" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41863 } }
I20260619 14:50:07.382589 6104 leader_election.cc:288] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [CANDIDATE]: Term 1 election: Requested vote from peers af1c76d650104af4b0f6940b31866e32 (127.0.241.3:38893), ac37b172aa254984b2a8fff585393998 (127.0.241.2:41863)
I20260619 14:50:07.383002 6034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b286ffd08e542f8b1b808dba5fdf0f8" candidate_uuid: "cce980fa59684e77ac8110ab15ee5cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af1c76d650104af4b0f6940b31866e32"
I20260619 14:50:07.383028 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b286ffd08e542f8b1b808dba5fdf0f8" candidate_uuid: "cce980fa59684e77ac8110ab15ee5cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac37b172aa254984b2a8fff585393998"
I20260619 14:50:07.383201 6034 raft_consensus.cc:3037] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:07.383215 5904 raft_consensus.cc:3037] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:07.386358 6034 raft_consensus.cc:2454] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cce980fa59684e77ac8110ab15ee5cc8 in term 1.
I20260619 14:50:07.386358 5904 raft_consensus.cc:2454] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cce980fa59684e77ac8110ab15ee5cc8 in term 1.
I20260619 14:50:07.386722 5708 leader_election.cc:302] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac37b172aa254984b2a8fff585393998, cce980fa59684e77ac8110ab15ee5cc8; no voters:
I20260619 14:50:07.387049 6104 raft_consensus.cc:2781] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:07.387490 6104 raft_consensus.cc:686] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 1 LEADER]: Becoming Leader. State: Replica: cce980fa59684e77ac8110ab15ee5cc8, State: Running, Role: LEADER
I20260619 14:50:07.387913 6104 consensus_queue.cc:237] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1c76d650104af4b0f6940b31866e32" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38893 } } peers { permanent_uuid: "cce980fa59684e77ac8110ab15ee5cc8" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45463 } } peers { permanent_uuid: "ac37b172aa254984b2a8fff585393998" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41863 } }
I20260619 14:50:07.392258 5631 catalog_manager.cc:5495] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to cce980fa59684e77ac8110ab15ee5cc8 (127.0.241.1). New cstate: current_term: 1 leader_uuid: "cce980fa59684e77ac8110ab15ee5cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af1c76d650104af4b0f6940b31866e32" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cce980fa59684e77ac8110ab15ee5cc8" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac37b172aa254984b2a8fff585393998" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41863 } health_report { overall_health: UNKNOWN } } }
W20260619 14:50:07.395210 5819 tablet.cc:2369] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:07.412729 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.410805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.0.241.1:45463)
0619 14:50:07.410967 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:07.410972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:07.411021 (+ 49us) server_negotiation.cc:407] Connection header received
0619 14:50:07.411198 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:07.411201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:07.411322 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:07.411407 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:07.412242 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:07.412246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:07.412249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:07.412263 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:07.412290 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:07.412301 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:07.412387 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:07.412494 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 14:50:07.412534 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 14:50:07.420284 6034 raft_consensus.cc:1261] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32 [term 1 FOLLOWER]: Refusing update from remote peer cce980fa59684e77ac8110ab15ee5cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:07.420284 5904 raft_consensus.cc:1261] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998 [term 1 FOLLOWER]: Refusing update from remote peer cce980fa59684e77ac8110ab15ee5cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:07.421218 6104 consensus_queue.cc:1035] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af1c76d650104af4b0f6940b31866e32" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:07.421476 6108 consensus_queue.cc:1035] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac37b172aa254984b2a8fff585393998" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:07.445536 5632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:07.445808 5632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:07.449527 5632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a411d392081485c9919a23c19d35292: Sending DeleteTablet for 3 replicas of tablet 2b286ffd08e542f8b1b808dba5fdf0f8
I20260619 14:50:07.450512 6014 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b286ffd08e542f8b1b808dba5fdf0f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:07 UTC) from {username='slave'} at 127.0.0.1:57780
I20260619 14:50:07.450580 5884 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b286ffd08e542f8b1b808dba5fdf0f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:07 UTC) from {username='slave'} at 127.0.0.1:48514
I20260619 14:50:07.450585 5753 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b286ffd08e542f8b1b808dba5fdf0f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:07 UTC) from {username='slave'} at 127.0.0.1:46070
I20260619 14:50:07.453583 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 5691
I20260619 14:50:07.461551 6121 tablet_replica.cc:330] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998: stopping tablet replica
I20260619 14:50:07.461751 6120 tablet_replica.cc:330] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32: stopping tablet replica
I20260619 14:50:07.461988 6121 raft_consensus.cc:2229] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:07.462085 6120 raft_consensus.cc:2229] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:07.462491 6121 raft_consensus.cc:2258] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:07.462487 6120 raft_consensus.cc:2258] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:07.465301 6120 ts_tablet_manager.cc:1905] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:07.467821 6122 tablet_replica.cc:330] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8: stopping tablet replica
I20260619 14:50:07.468228 6122 raft_consensus.cc:2229] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:07.468876 6122 raft_consensus.cc:2258] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:07.470830 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.455000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.0.241.1:45463)
0619 14:50:07.455187 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:07.455194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:07.468226 (+ 13032us) server_negotiation.cc:407] Connection header received
0619 14:50:07.468613 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:07.468620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:07.468686 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:07.468768 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:07.470059 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:07.470064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:07.470069 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:07.470087 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:07.470121 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:07.470135 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:07.470234 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:07.470473 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 14:50:07.470532 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:50:07.471207 6122 ts_tablet_manager.cc:1905] T 2b286ffd08e542f8b1b808dba5fdf0f8 P cce980fa59684e77ac8110ab15ee5cc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:07.472044 5813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:07.472915 6121 ts_tablet_manager.cc:1905] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:07.474659 6120 ts_tablet_manager.cc:1918] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:07.474823 6120 log.cc:1196] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/2b286ffd08e542f8b1b808dba5fdf0f8
I20260619 14:50:07.475466 6120 ts_tablet_manager.cc:1939] T 2b286ffd08e542f8b1b808dba5fdf0f8 P af1c76d650104af4b0f6940b31866e32: Deleting consensus metadata
I20260619 14:50:07.477381 5616 catalog_manager.cc:4841] TS af1c76d650104af4b0f6940b31866e32 (127.0.241.3:38893): tablet 2b286ffd08e542f8b1b808dba5fdf0f8 (table test-table [id=54ba47f9e69d4965bb3bd366e5560c35]) successfully deleted
I20260619 14:50:07.478900 6121 ts_tablet_manager.cc:1918] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:07.479117 6121 log.cc:1196] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/2b286ffd08e542f8b1b808dba5fdf0f8
I20260619 14:50:07.479682 6121 ts_tablet_manager.cc:1939] T 2b286ffd08e542f8b1b808dba5fdf0f8 P ac37b172aa254984b2a8fff585393998: Deleting consensus metadata
I20260619 14:50:07.481097 5618 catalog_manager.cc:4841] TS ac37b172aa254984b2a8fff585393998 (127.0.241.2:41863): tablet 2b286ffd08e542f8b1b808dba5fdf0f8 (table test-table [id=54ba47f9e69d4965bb3bd366e5560c35]) successfully deleted
I20260619 14:50:07.563709 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 5691
I20260619 14:50:07.584909 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 5822
W20260619 14:50:07.584909 5616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.241.1:45463 (error 108)
I20260619 14:50:07.586751 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.586282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45463 (local address 127.0.0.1:46170)
0619 14:50:07.586440 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.586571 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:50:07.587016 5616 catalog_manager.cc:4584] TS cce980fa59684e77ac8110ab15ee5cc8 (127.0.241.1:45463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b286ffd08e542f8b1b808dba5fdf0f8: Network error: Client connection negotiation failed: client connection to 127.0.241.1:45463: connect: Connection refused (error 111)
I20260619 14:50:07.587867 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.585657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.0.241.2:41863)
0619 14:50:07.585842 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:07.585848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:07.585877 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:50:07.586057 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:07.586063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:07.586156 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:07.586265 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:07.587182 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:07.587187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:07.587191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:07.587207 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:07.587242 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:07.587257 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:07.587403 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:07.587539 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 14:50:07.587596 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:50:07.588449 5944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:07.640535 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.640067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45463 (local address 127.0.0.1:46174)
0619 14:50:07.640283 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.640374 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 14:50:07.641628 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.641254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45463 (local address 127.0.0.1:46184)
0619 14:50:07.641408 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.641511 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 14:50:07.675827 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 5822
I20260619 14:50:07.692369 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 5952
I20260619 14:50:07.695235 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.693152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.0.241.3:38893)
0619 14:50:07.693360 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:07.693365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:07.693384 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:50:07.693443 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:07.693447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:07.693580 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:07.693687 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:07.694559 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:07.694564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:07.694567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:07.694582 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:07.694612 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:07.694624 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:07.694730 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:07.694825 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 14:50:07.694872 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 14:50:07.696000 6074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:07.701953 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.701551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45463 (local address 127.0.0.1:46186)
0619 14:50:07.701728 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.701821 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 14:50:07.702885 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.702581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45463 (local address 127.0.0.1:46188)
0619 14:50:07.702719 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.702785 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:50:07.784375 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.783874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45463 (local address 127.0.0.1:46198)
0619 14:50:07.784112 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.784228 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 14:50:07.784363 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 5952
I20260619 14:50:07.785616 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:07.785115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45463 (local address 127.0.0.1:46204)
0619 14:50:07.785319 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:07.785408 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:45463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 14:50:07.800648 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 5601
I20260619 14:50:07.801695 5662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:07.903790 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 5601
2026-06-19T14:50:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2494 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 14:50:07.943785 964 mini_oidc.cc:152] Starting JWKS server
I20260619 14:50:07.946241 964 webserver.cc:460] Webserver started at https://127.0.0.1:41459/ using document root <none> and password file <none>
I20260619 14:50:07.946542 964 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:50:07.947126 964 webserver.cc:460] Webserver started at http://127.0.0.1:40601/ using document root <none> and password file <none>
2026-06-19T14:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:07Z Disabled control of system clock
I20260619 14:50:07.972537 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33449
--webserver_interface=127.0.241.62
--webserver_port=0
--builtin_ntp_servers=127.0.241.20:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:33449
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41459/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/testchainca.cert with env {}
W20260619 14:50:08.297070 6134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:08.297354 6134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:08.297446 6134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:08.304872 6134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:08.304988 6134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:08.305032 6134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:08.305095 6134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:08.305148 6134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:50:08.305209 6134 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:50:08.305255 6134 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41459/jwks.json
I20260619 14:50:08.315258 6134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:33449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41459/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:08.316579 6134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:08.317925 6134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:08.330221 6139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:08.330180 6142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:08.330147 6140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:08.331485 6134 server_base.cc:1034] running on GCE node
I20260619 14:50:08.332269 6134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:08.333734 6134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:08.335095 6134 hybrid_clock.cc:648] HybridClock initialized: now 1781880608335029 us; error 55 us; skew 500 ppm
I20260619 14:50:08.335518 6134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:08.343403 6134 webserver.cc:460] Webserver started at http://127.0.241.62:36597/ using document root <none> and password file <none>
I20260619 14:50:08.344192 6134 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:08.344332 6134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:08.344643 6134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:08.348181 6134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "51fc86a63e8748ae8cf07b40cd8ea5bf"
format_stamp: "Formatted at 2026-06-19 14:50:08 on dist-test-slave-p3ml"
server_key: "d3f12bc3afdf6a074e6d2ea347aebe6d"
server_key_iv: "1a5cf793679a2173ba035ce725237d2c"
server_key_version: "encryptionkey@0"
I20260619 14:50:08.349046 6134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "51fc86a63e8748ae8cf07b40cd8ea5bf"
format_stamp: "Formatted at 2026-06-19 14:50:08 on dist-test-slave-p3ml"
server_key: "d3f12bc3afdf6a074e6d2ea347aebe6d"
server_key_iv: "1a5cf793679a2173ba035ce725237d2c"
server_key_version: "encryptionkey@0"
I20260619 14:50:08.354758 6134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:50:08.358345 6148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:08.359786 6134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:50:08.359972 6134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "51fc86a63e8748ae8cf07b40cd8ea5bf"
format_stamp: "Formatted at 2026-06-19 14:50:08 on dist-test-slave-p3ml"
server_key: "d3f12bc3afdf6a074e6d2ea347aebe6d"
server_key_iv: "1a5cf793679a2173ba035ce725237d2c"
server_key_version: "encryptionkey@0"
I20260619 14:50:08.360183 6134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:08.377591 6134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:08.378355 6134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:08.378659 6134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:08.398888 6134 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:33449
I20260619 14:50:08.398907 6199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:33449 every 8 connection(s)
I20260619 14:50:08.400228 6134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:08.403770 6200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:08.409031 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 6134
I20260619 14:50:08.409250 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:08.409962 964 external_mini_cluster.cc:1428] Setting key f9db01e985f5402d644704896d849447
I20260619 14:50:08.414160 6200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf: Bootstrap starting.
I20260619 14:50:08.417385 6200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:08.418532 6200 log.cc:824] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:08.421520 6200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf: No bootstrap required, opened a new log
I20260619 14:50:08.424170 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:08.411882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.0.241.62:33449)
0619 14:50:08.412922 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:08.412932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:08.413004 (+ 72us) server_negotiation.cc:407] Connection header received
0619 14:50:08.414889 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:08.414910 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:08.415339 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:08.415753 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:08.416388 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:08.417779 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:08.419627 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:08.420351 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:08.420528 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:08.421780 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:08.421788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:08.421802 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:08.422356 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:08.422438 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:08.422461 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:08.422561 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:08.422809 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 14:50:08.423151 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":412,"threads_started":1}
I20260619 14:50:08.427691 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:50:08.429181 6200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc86a63e8748ae8cf07b40cd8ea5bf" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33449 } }
I20260619 14:50:08.429513 6200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:08.429611 6200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fc86a63e8748ae8cf07b40cd8ea5bf, State: Initialized, Role: FOLLOWER
I20260619 14:50:08.430197 6200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc86a63e8748ae8cf07b40cd8ea5bf" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33449 } }
I20260619 14:50:08.430426 6200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:08.430567 6200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:08.430733 6200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:08.432804 6200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc86a63e8748ae8cf07b40cd8ea5bf" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33449 } }
I20260619 14:50:08.433252 6200 leader_election.cc:302] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51fc86a63e8748ae8cf07b40cd8ea5bf; no voters:
I20260619 14:50:08.433789 6200 leader_election.cc:288] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:08.433943 6205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:08.434705 6205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [term 1 LEADER]: Becoming Leader. State: Replica: 51fc86a63e8748ae8cf07b40cd8ea5bf, State: Running, Role: LEADER
I20260619 14:50:08.435226 6205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc86a63e8748ae8cf07b40cd8ea5bf" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33449 } }
I20260619 14:50:08.435745 6200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:08.444109 6206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51fc86a63e8748ae8cf07b40cd8ea5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc86a63e8748ae8cf07b40cd8ea5bf" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33449 } } }
I20260619 14:50:08.444105 6207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51fc86a63e8748ae8cf07b40cd8ea5bf. Latest consensus state: current_term: 1 leader_uuid: "51fc86a63e8748ae8cf07b40cd8ea5bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51fc86a63e8748ae8cf07b40cd8ea5bf" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33449 } } }
I20260619 14:50:08.444540 6206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:08.444585 6207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:08.445757 6219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:08.450680 6219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:08.458385 6219 catalog_manager.cc:1347] Generated new cluster ID: f4acef5700684a24bdc089420a41f6c9
I20260619 14:50:08.458509 6219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:08.471676 6219 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:08.472886 6219 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:08.482332 6219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51fc86a63e8748ae8cf07b40cd8ea5bf: Generated new TSK 0
I20260619 14:50:08.483139 6219 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:50:08.525336 6230 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 14:50:08.525527 6230 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:50:08.526302 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:08.497148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.0.241.62:33449)
0619 14:50:08.497391 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:08.497399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:08.497798 (+ 399us) server_negotiation.cc:407] Connection header received
0619 14:50:08.498095 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:08.498099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:08.498173 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:08.498296 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:50:08.498969 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:08.499751 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:08.501816 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:08.502263 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:08.502351 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:08.502952 (+ 601us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:50:08.525797 (+ 22845us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41459/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:50:08.526088 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41304: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41459/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 14:50:08.526444 6203 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41304: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41459/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 14:50:08.528434 6229 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:08.497029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33449 (local address 127.0.0.1:41304)
0619 14:50:08.497691 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:08.497711 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:08.497889 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:08.498362 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:08.498367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:08.498381 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:50:08.498809 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:08.498826 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:08.499890 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:08.499896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:08.501684 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:08.501700 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:08.502444 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:08.502451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:08.502645 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:08.502720 (+ 75us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:50:08.526093 (+ 23373us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41459/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:50:08.526440 (+ 347us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.241.62:33449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41459/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":324,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.241.62:33449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41459/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 14:50:08.530368 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 6134
I20260619 14:50:08.531564 6195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:08.621968 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 6134
2026-06-19T14:50:08Z chronyd exiting
I20260619 14:50:08.672148 964 test_util.cc:175] -----------------------------------------------
I20260619 14:50:08.672288 964 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880565277815-964-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (732 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 14:50:08.675995 964 mini_oidc.cc:152] Starting JWKS server
I20260619 14:50:08.677790 964 webserver.cc:460] Webserver started at https://127.0.0.1:39049/ using document root <none> and password file <none>
I20260619 14:50:08.678051 964 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:50:08.678678 964 webserver.cc:460] Webserver started at http://127.0.0.1:37325/ using document root <none> and password file <none>
2026-06-19T14:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:08Z Disabled control of system clock
I20260619 14:50:08.703652 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:38429
--webserver_interface=127.0.241.62
--webserver_port=0
--builtin_ntp_servers=127.0.241.20:35053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:38429
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39049/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/testchainca.cert with env {}
W20260619 14:50:09.011806 6237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:09.012089 6237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:09.012174 6237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:09.019840 6237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:09.019958 6237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:09.020010 6237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:09.020089 6237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:09.020138 6237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:50:09.020207 6237 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:50:09.020253 6237 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39049/jwks.json
I20260619 14:50:09.030651 6237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:35053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:38429
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:38429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39049/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:09.031945 6237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:09.033155 6237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:09.045181 6245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:09.045223 6243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:09.045156 6242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:09.049114 6237 server_base.cc:1034] running on GCE node
I20260619 14:50:09.050047 6237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:09.051638 6237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:09.052891 6237 hybrid_clock.cc:648] HybridClock initialized: now 1781880609052840 us; error 56 us; skew 500 ppm
I20260619 14:50:09.053308 6237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:09.055805 6237 webserver.cc:460] Webserver started at http://127.0.241.62:42311/ using document root <none> and password file <none>
I20260619 14:50:09.056564 6237 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:09.056715 6237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:09.057070 6237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:09.060181 6237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "faa0db9dc3a646d9abd072b941ca5a6c"
format_stamp: "Formatted at 2026-06-19 14:50:09 on dist-test-slave-p3ml"
server_key: "f0e6f714a9718c515106f6186e9bb1bb"
server_key_iv: "803f76f9fe2602e94f01b881badb2133"
server_key_version: "encryptionkey@0"
I20260619 14:50:09.061058 6237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "faa0db9dc3a646d9abd072b941ca5a6c"
format_stamp: "Formatted at 2026-06-19 14:50:09 on dist-test-slave-p3ml"
server_key: "f0e6f714a9718c515106f6186e9bb1bb"
server_key_iv: "803f76f9fe2602e94f01b881badb2133"
server_key_version: "encryptionkey@0"
I20260619 14:50:09.066306 6237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 14:50:09.069720 6251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:09.071044 6237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:50:09.071247 6237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "faa0db9dc3a646d9abd072b941ca5a6c"
format_stamp: "Formatted at 2026-06-19 14:50:09 on dist-test-slave-p3ml"
server_key: "f0e6f714a9718c515106f6186e9bb1bb"
server_key_iv: "803f76f9fe2602e94f01b881badb2133"
server_key_version: "encryptionkey@0"
I20260619 14:50:09.071485 6237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:09.099390 6237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:09.100314 6237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:09.100629 6237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:09.118894 6237 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:38429
I20260619 14:50:09.118901 6302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:38429 every 8 connection(s)
I20260619 14:50:09.120265 6237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:09.123948 6303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:09.130088 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 6237
I20260619 14:50:09.130312 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:09.131142 964 external_mini_cluster.cc:1428] Setting key daccdd3e835ba67b7b2cdc3244b19b91
I20260619 14:50:09.135084 6303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c: Bootstrap starting.
I20260619 14:50:09.138141 6303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:09.139214 6303 log.cc:824] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:09.141860 6303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c: No bootstrap required, opened a new log
I20260619 14:50:09.145135 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:09.133210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.0.241.62:38429)
0619 14:50:09.134249 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:09.134260 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:09.134306 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:50:09.136233 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:09.136255 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:09.136768 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:09.137171 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:09.137762 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:09.139194 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:09.140572 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:09.141291 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:09.141551 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:09.142808 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:09.142816 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:09.142829 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:09.143369 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:09.143500 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:09.143532 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:09.143673 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:09.143892 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 14:50:09.144197 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":381,"threads_started":1}
I20260619 14:50:09.149021 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:50:09.150244 6303 raft_consensus.cc:348] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa0db9dc3a646d9abd072b941ca5a6c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38429 } }
I20260619 14:50:09.150579 6303 raft_consensus.cc:374] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:09.150677 6303 raft_consensus.cc:729] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa0db9dc3a646d9abd072b941ca5a6c, State: Initialized, Role: FOLLOWER
I20260619 14:50:09.151309 6303 consensus_queue.cc:260] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa0db9dc3a646d9abd072b941ca5a6c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38429 } }
I20260619 14:50:09.151597 6303 raft_consensus.cc:388] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:09.151742 6303 raft_consensus.cc:482] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:09.151919 6303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:09.154188 6303 raft_consensus.cc:504] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa0db9dc3a646d9abd072b941ca5a6c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38429 } }
I20260619 14:50:09.154687 6303 leader_election.cc:302] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faa0db9dc3a646d9abd072b941ca5a6c; no voters:
I20260619 14:50:09.155347 6303 leader_election.cc:288] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:09.155494 6308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:09.156275 6308 raft_consensus.cc:686] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [term 1 LEADER]: Becoming Leader. State: Replica: faa0db9dc3a646d9abd072b941ca5a6c, State: Running, Role: LEADER
I20260619 14:50:09.156831 6308 consensus_queue.cc:237] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa0db9dc3a646d9abd072b941ca5a6c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38429 } }
I20260619 14:50:09.157568 6303 sys_catalog.cc:564] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:09.159730 6310 sys_catalog.cc:455] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader faa0db9dc3a646d9abd072b941ca5a6c. Latest consensus state: current_term: 1 leader_uuid: "faa0db9dc3a646d9abd072b941ca5a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa0db9dc3a646d9abd072b941ca5a6c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38429 } } }
I20260619 14:50:09.160157 6310 sys_catalog.cc:458] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:09.159921 6309 sys_catalog.cc:455] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faa0db9dc3a646d9abd072b941ca5a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa0db9dc3a646d9abd072b941ca5a6c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38429 } } }
I20260619 14:50:09.160493 6309 sys_catalog.cc:458] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:09.168023 6313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:09.173372 6313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:09.181159 6313 catalog_manager.cc:1347] Generated new cluster ID: 2471fd9a5b0540c1a48ebabfdfba913f
I20260619 14:50:09.181267 6313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:09.211486 6313 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:09.213302 6313 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:09.221156 6313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faa0db9dc3a646d9abd072b941ca5a6c: Generated new TSK 0
I20260619 14:50:09.222193 6313 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:50:09.261171 6333 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 14:50:09.261307 6333 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:50:09.262485 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:09.242121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.0.241.62:38429)
0619 14:50:09.242418 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:09.242427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:09.243129 (+ 702us) server_negotiation.cc:407] Connection header received
0619 14:50:09.243458 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:09.243467 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:09.243565 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:09.243762 (+ 197us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:50:09.244441 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:09.245683 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:09.247029 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:09.247570 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:09.247711 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:09.248157 (+ 446us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:50:09.261798 (+ 13641us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:50:09.262187 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42672: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 14:50:09.262611 6306 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42672: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 14:50:09.262669 6332 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:09.241855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:38429 (local address 127.0.0.1:42672)
0619 14:50:09.242992 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:09.243028 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:09.243177 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:09.243799 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:09.243805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:09.243819 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:50:09.244198 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:09.244219 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:09.245826 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:09.245831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:09.246893 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:09.246904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:09.247722 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:09.247727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:09.247884 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:09.247915 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:50:09.262020 (+ 14105us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:50:09.262286 (+ 266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.241.62:38429: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":752,"threads_started":1}
I20260619 14:50:09.263972 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 6237
I20260619 14:50:09.265193 6298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:09.357604 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 6237
2026-06-19T14:50:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (737 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 14:50:09.413739 964 mini_oidc.cc:152] Starting JWKS server
I20260619 14:50:09.415768 964 webserver.cc:460] Webserver started at https://127.0.0.1:35451/ using document root <none> and password file <none>
I20260619 14:50:09.416020 964 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:50:09.416735 964 webserver.cc:460] Webserver started at http://127.0.0.1:34669/ using document root <none> and password file <none>
2026-06-19T14:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:09Z Disabled control of system clock
I20260619 14:50:09.442785 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35513
--webserver_interface=127.0.241.62
--webserver_port=0
--builtin_ntp_servers=127.0.241.20:40707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:35513
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35451/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:50:09.766132 6340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:09.766435 6340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:09.766530 6340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:09.773492 6340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:09.773591 6340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:09.773629 6340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:09.773681 6340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:09.773718 6340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:50:09.773778 6340 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:50:09.773819 6340 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35451/jwks.json
I20260619 14:50:09.782636 6340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:35513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35451/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:09.783831 6340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:09.785496 6340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:09.796871 6345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:09.796799 6348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:09.796902 6346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:09.797683 6340 server_base.cc:1034] running on GCE node
I20260619 14:50:09.798517 6340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:09.799871 6340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:09.801054 6340 hybrid_clock.cc:648] HybridClock initialized: now 1781880609801000 us; error 53 us; skew 500 ppm
I20260619 14:50:09.801328 6340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:09.803385 6340 webserver.cc:460] Webserver started at http://127.0.241.62:44703/ using document root <none> and password file <none>
I20260619 14:50:09.804071 6340 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:09.804152 6340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:09.804412 6340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:09.807054 6340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "8a18647f6ce84ec78ed5666d40e3a354"
format_stamp: "Formatted at 2026-06-19 14:50:09 on dist-test-slave-p3ml"
server_key: "27afda44985b6b84524f6c6dc2a6a0fd"
server_key_iv: "729383e40c34c1c4f1a91d9763cd5c0a"
server_key_version: "encryptionkey@0"
I20260619 14:50:09.807794 6340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "8a18647f6ce84ec78ed5666d40e3a354"
format_stamp: "Formatted at 2026-06-19 14:50:09 on dist-test-slave-p3ml"
server_key: "27afda44985b6b84524f6c6dc2a6a0fd"
server_key_iv: "729383e40c34c1c4f1a91d9763cd5c0a"
server_key_version: "encryptionkey@0"
I20260619 14:50:09.812482 6340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:50:09.815548 6354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:09.816951 6340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:50:09.817165 6340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "8a18647f6ce84ec78ed5666d40e3a354"
format_stamp: "Formatted at 2026-06-19 14:50:09 on dist-test-slave-p3ml"
server_key: "27afda44985b6b84524f6c6dc2a6a0fd"
server_key_iv: "729383e40c34c1c4f1a91d9763cd5c0a"
server_key_version: "encryptionkey@0"
I20260619 14:50:09.817360 6340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:09.832587 6340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:09.833389 6340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:09.833726 6340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:09.853035 6340 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:35513
I20260619 14:50:09.853057 6405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:35513 every 8 connection(s)
I20260619 14:50:09.854439 6340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:09.858088 6406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:09.859798 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 6340
I20260619 14:50:09.859989 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:09.860620 964 external_mini_cluster.cc:1428] Setting key 0d85f06eb27141ae78654647e88c8ad7
I20260619 14:50:09.869503 6406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354: Bootstrap starting.
I20260619 14:50:09.873099 6406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:09.873953 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:09.862550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40916 (local address 127.0.241.62:35513)
0619 14:50:09.863654 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:09.863663 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:09.863705 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:50:09.865250 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:09.865268 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:09.865659 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:09.865973 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:09.866571 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:09.867914 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:09.869749 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:09.870371 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:09.870492 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:09.871760 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:09.871767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:09.871779 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:09.872267 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:09.872353 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:09.872372 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:09.872454 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:09.872692 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 14:50:09.873033 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":403,"threads_started":1}
I20260619 14:50:09.874199 6406 log.cc:824] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:09.877266 6406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354: No bootstrap required, opened a new log
I20260619 14:50:09.877683 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:50:09.884971 6406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a18647f6ce84ec78ed5666d40e3a354" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35513 } }
I20260619 14:50:09.885339 6406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:09.885449 6406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a18647f6ce84ec78ed5666d40e3a354, State: Initialized, Role: FOLLOWER
I20260619 14:50:09.885984 6406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a18647f6ce84ec78ed5666d40e3a354" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35513 } }
I20260619 14:50:09.886215 6406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:09.886353 6406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:09.886513 6406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:09.888655 6406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a18647f6ce84ec78ed5666d40e3a354" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35513 } }
I20260619 14:50:09.889129 6406 leader_election.cc:302] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a18647f6ce84ec78ed5666d40e3a354; no voters:
I20260619 14:50:09.889788 6406 leader_election.cc:288] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:09.889911 6411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:09.890556 6411 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [term 1 LEADER]: Becoming Leader. State: Replica: 8a18647f6ce84ec78ed5666d40e3a354, State: Running, Role: LEADER
I20260619 14:50:09.891062 6411 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a18647f6ce84ec78ed5666d40e3a354" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35513 } }
I20260619 14:50:09.892060 6406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:09.893896 6412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a18647f6ce84ec78ed5666d40e3a354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a18647f6ce84ec78ed5666d40e3a354" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35513 } } }
I20260619 14:50:09.894362 6412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:09.894272 6413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a18647f6ce84ec78ed5666d40e3a354. Latest consensus state: current_term: 1 leader_uuid: "8a18647f6ce84ec78ed5666d40e3a354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a18647f6ce84ec78ed5666d40e3a354" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35513 } } }
I20260619 14:50:09.894770 6413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:09.900100 6418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:09.906636 6418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:09.916190 6418 catalog_manager.cc:1347] Generated new cluster ID: 36403cceba8946b0b9405232b04f6cff
I20260619 14:50:09.916306 6418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:09.926545 6418 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:09.927848 6418 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:09.934405 6418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a18647f6ce84ec78ed5666d40e3a354: Generated new TSK 0
I20260619 14:50:09.935148 6418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:50:09.961012 6409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:09.952778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.0.241.62:35513)
0619 14:50:09.952952 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:09.952958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:09.953355 (+ 397us) server_negotiation.cc:407] Connection header received
0619 14:50:09.953531 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:09.953536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:09.953622 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:09.953793 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:50:09.954207 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:09.955354 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:09.956738 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:09.957453 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:09.957576 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:09.958072 (+ 496us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:50:09.960420 (+ 2348us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35451/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:50:09.960718 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35451/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 14:50:09.961151 6409 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35451/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 14:50:09.962807 6435 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:09.952660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35513 (local address 127.0.0.1:40932)
0619 14:50:09.953252 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:09.953268 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:09.953379 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:09.953804 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:09.953807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:09.953815 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:50:09.954068 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:09.954075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:09.955572 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:09.955575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:09.956602 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:09.956614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:09.957648 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:09.957653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:09.957784 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:09.957813 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:50:09.960637 (+ 2824us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35451/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:50:09.962398 (+ 1761us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.241.62:35513: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35451/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":1493,"thread_start_us":318,"threads_started":1}
I20260619 14:50:09.964480 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 6340
I20260619 14:50:09.965698 6401 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 14:50:09.973740 6436 webserver.cc:533] Webserver: sslize failed:
E20260619 14:50:09.973825 6436 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:50:10.056833 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 6340
2026-06-19T14:50:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (698 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T14:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:10Z Disabled control of system clock
I20260619 14:50:10.137653 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35853
--webserver_interface=127.0.241.62
--webserver_port=0
--builtin_ntp_servers=127.0.241.20:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:35853
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/testchainca.cert with env {}
W20260619 14:50:10.458619 6441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:10.458922 6441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:10.459025 6441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:10.466233 6441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:10.466357 6441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:10.466403 6441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:10.466466 6441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:10.466521 6441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:50:10.466584 6441 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:50:10.466631 6441 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 14:50:10.476650 6441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:35853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:10.477967 6441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:10.479382 6441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:10.491645 6447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:10.491524 6449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:10.491496 6446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:10.492976 6441 server_base.cc:1034] running on GCE node
I20260619 14:50:10.493834 6441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:10.495170 6441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:10.496387 6441 hybrid_clock.cc:648] HybridClock initialized: now 1781880610496334 us; error 71 us; skew 500 ppm
I20260619 14:50:10.496733 6441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:10.498914 6441 webserver.cc:460] Webserver started at http://127.0.241.62:36001/ using document root <none> and password file <none>
I20260619 14:50:10.499686 6441 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:10.499818 6441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:10.500125 6441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:10.503125 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "159943f45f3d461db03946bba24a1b27"
format_stamp: "Formatted at 2026-06-19 14:50:10 on dist-test-slave-p3ml"
server_key: "f6da83f7802d3840e391e5f8b98f2ff5"
server_key_iv: "cf2f68b577770d3132d11f969a28f0f5"
server_key_version: "encryptionkey@0"
I20260619 14:50:10.503999 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "159943f45f3d461db03946bba24a1b27"
format_stamp: "Formatted at 2026-06-19 14:50:10 on dist-test-slave-p3ml"
server_key: "f6da83f7802d3840e391e5f8b98f2ff5"
server_key_iv: "cf2f68b577770d3132d11f969a28f0f5"
server_key_version: "encryptionkey@0"
I20260619 14:50:10.509042 6441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:50:10.512463 6455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:10.513749 6441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:50:10.513917 6441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "159943f45f3d461db03946bba24a1b27"
format_stamp: "Formatted at 2026-06-19 14:50:10 on dist-test-slave-p3ml"
server_key: "f6da83f7802d3840e391e5f8b98f2ff5"
server_key_iv: "cf2f68b577770d3132d11f969a28f0f5"
server_key_version: "encryptionkey@0"
I20260619 14:50:10.514123 6441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:10.545627 6441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:10.546481 6441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:10.546787 6441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:10.564845 6441 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:35853
I20260619 14:50:10.564864 6506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:35853 every 8 connection(s)
I20260619 14:50:10.566195 6441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:10.570070 6507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:10.574949 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 6441
I20260619 14:50:10.575165 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:10.575845 964 external_mini_cluster.cc:1428] Setting key dcf0a9ddaa07126ac9bbcfd293a505df
I20260619 14:50:10.580875 6507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27: Bootstrap starting.
I20260619 14:50:10.584471 6507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:10.585675 6507 log.cc:824] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:10.588496 6507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27: No bootstrap required, opened a new log
I20260619 14:50:10.589742 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.577855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.0.241.62:35853)
0619 14:50:10.578922 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:10.578932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:10.578976 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:50:10.580635 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:10.580657 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:10.581195 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:10.581525 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:10.582153 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.583531 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.585345 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.586084 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.586318 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.587423 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:10.587476 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:10.587493 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:10.588022 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:10.588103 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:10.588121 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:10.588241 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:10.588477 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 14:50:10.588848 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":417,"threads_started":1}
I20260619 14:50:10.593739 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:50:10.598040 6507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159943f45f3d461db03946bba24a1b27" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35853 } }
I20260619 14:50:10.598363 6507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:10.598433 6507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 159943f45f3d461db03946bba24a1b27, State: Initialized, Role: FOLLOWER
I20260619 14:50:10.598891 6507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159943f45f3d461db03946bba24a1b27" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35853 } }
I20260619 14:50:10.599079 6507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:10.599176 6507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:10.599289 6507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:10.601358 6507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159943f45f3d461db03946bba24a1b27" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35853 } }
I20260619 14:50:10.601956 6507 leader_election.cc:302] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 159943f45f3d461db03946bba24a1b27; no voters:
I20260619 14:50:10.602572 6507 leader_election.cc:288] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:10.602766 6512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:10.603510 6512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [term 1 LEADER]: Becoming Leader. State: Replica: 159943f45f3d461db03946bba24a1b27, State: Running, Role: LEADER
I20260619 14:50:10.604089 6512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159943f45f3d461db03946bba24a1b27" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35853 } }
I20260619 14:50:10.604641 6507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:10.606652 6514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 159943f45f3d461db03946bba24a1b27. Latest consensus state: current_term: 1 leader_uuid: "159943f45f3d461db03946bba24a1b27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159943f45f3d461db03946bba24a1b27" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35853 } } }
I20260619 14:50:10.607008 6514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:10.607515 6513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "159943f45f3d461db03946bba24a1b27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "159943f45f3d461db03946bba24a1b27" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35853 } } }
I20260619 14:50:10.607868 6513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:10.611522 6517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:10.618278 6517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:10.626817 6517 catalog_manager.cc:1347] Generated new cluster ID: 7c5076136a9048b4b8702f6390e5764e
I20260619 14:50:10.626936 6517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:10.642045 6517 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:10.643952 6517 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:10.653597 6517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 159943f45f3d461db03946bba24a1b27: Generated new TSK 0
I20260619 14:50:10.654623 6517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:50:10.700970 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.677509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.0.241.62:35853)
0619 14:50:10.677708 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:10.677714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:10.678038 (+ 324us) server_negotiation.cc:407] Connection header received
0619 14:50:10.678256 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:10.678261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:10.678358 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:10.678520 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:50:10.678990 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.680169 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.681739 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.682442 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.682585 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.683098 (+ 513us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:50:10.700231 (+ 17133us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:50:10.700646 (+ 415us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57374: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 14:50:10.701125 6510 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57374: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 14:50:10.701129 6536 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:10.677322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35853 (local address 127.0.0.1:57374)
0619 14:50:10.677918 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:10.677933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:10.678069 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:10.678557 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:10.678561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:10.678570 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:50:10.678838 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.678846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.680335 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.680339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.681596 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.681607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.682666 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.682672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.682805 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.682840 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:50:10.700538 (+ 17698us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:50:10.700779 (+ 241us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.241.62:35853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":310,"threads_started":1}
I20260619 14:50:10.702531 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 6441
I20260619 14:50:10.703670 6502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:10.795126 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 6441
2026-06-19T14:50:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (717 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (394 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (404 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:50:11 dist-test-slave-p3ml krb5kdc[6551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:50:11 dist-test-slave-p3ml krb5kdc[6551](info): set up 2 sockets
Jun 19 14:50:11 dist-test-slave-p3ml krb5kdc[6551](info): commencing operation
krb5kdc: starting...
W20260619 14:50:12.668100 964 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:50:12 dist-test-slave-p3ml krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:50:12.863126 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:36715
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:36715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:13.188454 6564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:13.188814 6564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:13.188984 6564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:13.196280 6564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:13.196449 6564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:13.196584 6564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:13.196703 6564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:13.196800 6564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:13.206646 6564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:36715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:36715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:13.208019 6564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:13.209376 6564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:13.221460 6570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:13.221431 6569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:13.221498 6572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:13.222316 6564 server_base.cc:1034] running on GCE node
I20260619 14:50:13.223004 6564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:13.224426 6564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:13.225652 6564 hybrid_clock.cc:648] HybridClock initialized: now 1781880613225611 us; error 55 us; skew 500 ppm
I20260619 14:50:13.225996 6564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:13 dist-test-slave-p3ml krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:13.229206 6564 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:50:13.230711 6564 webserver.cc:460] Webserver started at http://127.0.241.62:45387/ using document root <none> and password file <none>
I20260619 14:50:13.231405 6564 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:13.231608 6564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:13.231925 6564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:13.234759 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "971a96f896b242478210b7b500dec4fa"
format_stamp: "Formatted at 2026-06-19 14:50:13 on dist-test-slave-p3ml"
server_key: "0a3051b90b5bd53db008d3cdd3f19795"
server_key_iv: "faa03bb9fb1ccca83829657f58073be2"
server_key_version: "encryptionkey@0"
I20260619 14:50:13.235592 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "971a96f896b242478210b7b500dec4fa"
format_stamp: "Formatted at 2026-06-19 14:50:13 on dist-test-slave-p3ml"
server_key: "0a3051b90b5bd53db008d3cdd3f19795"
server_key_iv: "faa03bb9fb1ccca83829657f58073be2"
server_key_version: "encryptionkey@0"
I20260619 14:50:13.240448 6564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:50:13.243834 6579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:13.245110 6564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:50:13.245304 6564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "971a96f896b242478210b7b500dec4fa"
format_stamp: "Formatted at 2026-06-19 14:50:13 on dist-test-slave-p3ml"
server_key: "0a3051b90b5bd53db008d3cdd3f19795"
server_key_iv: "faa03bb9fb1ccca83829657f58073be2"
server_key_version: "encryptionkey@0"
I20260619 14:50:13.245501 6564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:13.259380 6564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:13.264098 6564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:13.264400 6564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:13.282222 6564 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:36715
I20260619 14:50:13.282245 6630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:36715 every 8 connection(s)
I20260619 14:50:13.283668 6564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:13.287245 6631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:13.292301 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 6564
I20260619 14:50:13.292527 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:13.293243 964 external_mini_cluster.cc:1428] Setting key 201a7b932171ff179a22f9e7f9dbbdbf
I20260619 14:50:13.298375 6631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa: Bootstrap starting.
I20260619 14:50:13.301369 6631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:13.302484 6631 log.cc:824] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:13.305869 6631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa: No bootstrap required, opened a new log
Jun 19 14:50:13 dist-test-slave-p3ml krb5kdc[6551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:13.313525 6631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971a96f896b242478210b7b500dec4fa" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36715 } }
I20260619 14:50:13.313882 6631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:13.314005 6631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971a96f896b242478210b7b500dec4fa, State: Initialized, Role: FOLLOWER
I20260619 14:50:13.314620 6631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971a96f896b242478210b7b500dec4fa" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36715 } }
I20260619 14:50:13.314849 6631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:13.314988 6631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:13.315171 6631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:13.315834 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:13.295295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.0.241.62:36715)
0619 14:50:13.296155 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:13.296169 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:13.296213 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:50:13.297863 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:13.297884 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:13.298256 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:13.298706 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:13.300168 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:13.301408 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:13.302670 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:13.303735 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:13.303909 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:13.308568 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:13.308575 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:13.308591 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:13.308626 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:13.311711 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:13.312407 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:13.312413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:13.312419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:13.312574 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:13.312942 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:13.312949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:13.312952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:13.313439 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:13.313753 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:13.314167 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 14:50:13.314584 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":437,"threads_started":1}
I20260619 14:50:13.317526 6631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971a96f896b242478210b7b500dec4fa" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36715 } }
I20260619 14:50:13.318002 6631 leader_election.cc:302] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 971a96f896b242478210b7b500dec4fa; no voters:
I20260619 14:50:13.318645 6631 leader_election.cc:288] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:13.318739 6636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:13.319375 6636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [term 1 LEADER]: Becoming Leader. State: Replica: 971a96f896b242478210b7b500dec4fa, State: Running, Role: LEADER
I20260619 14:50:13.319952 6636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971a96f896b242478210b7b500dec4fa" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36715 } }
I20260619 14:50:13.320806 6631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:13.322788 6638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 971a96f896b242478210b7b500dec4fa. Latest consensus state: current_term: 1 leader_uuid: "971a96f896b242478210b7b500dec4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971a96f896b242478210b7b500dec4fa" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36715 } } }
I20260619 14:50:13.323132 6638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:13.322978 6637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "971a96f896b242478210b7b500dec4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971a96f896b242478210b7b500dec4fa" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36715 } } }
I20260619 14:50:13.323340 6637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:13.327150 6643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:13.332455 6643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:13.340885 6643 catalog_manager.cc:1347] Generated new cluster ID: 3223b3c1dc0746b58fa7529adc427fe4
I20260619 14:50:13.341007 6643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:13.355746 6643 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:13.356902 6643 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:13.370303 6643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 971a96f896b242478210b7b500dec4fa: Generated new TSK 0
I20260619 14:50:13.371094 6643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:50:13.434825 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:36715
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:13.773916 6657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:13.774168 6657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:13.774376 6657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:13.781469 6657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:13.781601 6657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:13.781788 6657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:50:13.791236 6657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:36715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:13.792439 6657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:13.793725 6657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:13.806471 6662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:13.806516 6665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:13.806430 6663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:13.807730 6657 server_base.cc:1034] running on GCE node
I20260619 14:50:13.808282 6657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:13.809125 6657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:13.810333 6657 hybrid_clock.cc:648] HybridClock initialized: now 1781880613810284 us; error 35 us; skew 500 ppm
I20260619 14:50:13.810633 6657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:13 dist-test-slave-p3ml krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:13.818622 6657 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:50:13.820024 6657 webserver.cc:460] Webserver started at http://127.0.241.1:45723/ using document root <none> and password file <none>
I20260619 14:50:13.820667 6657 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:13.820760 6657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:13.821094 6657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:13.823856 6657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "3864c241055745458a2613ccad2ef715"
format_stamp: "Formatted at 2026-06-19 14:50:13 on dist-test-slave-p3ml"
server_key: "18192be48b77375ba6ad3a4019d7ebf9"
server_key_iv: "347978fadd9a18d676b0532e150d7383"
server_key_version: "encryptionkey@0"
I20260619 14:50:13.824605 6657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "3864c241055745458a2613ccad2ef715"
format_stamp: "Formatted at 2026-06-19 14:50:13 on dist-test-slave-p3ml"
server_key: "18192be48b77375ba6ad3a4019d7ebf9"
server_key_iv: "347978fadd9a18d676b0532e150d7383"
server_key_version: "encryptionkey@0"
I20260619 14:50:13.829645 6657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:50:13.832818 6672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:13.834081 6657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:50:13.834200 6657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "3864c241055745458a2613ccad2ef715"
format_stamp: "Formatted at 2026-06-19 14:50:13 on dist-test-slave-p3ml"
server_key: "18192be48b77375ba6ad3a4019d7ebf9"
server_key_iv: "347978fadd9a18d676b0532e150d7383"
server_key_version: "encryptionkey@0"
I20260619 14:50:13.834355 6657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:13.848917 6657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:13.852886 6657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:13.853111 6657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:13.853925 6657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:13.855103 6657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:13.855173 6657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:13.855253 6657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:13.855296 6657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:13.886643 6657 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:32927
I20260619 14:50:13.886732 6784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:32927 every 8 connection(s)
I20260619 14:50:13.887986 6657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:13.896404 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 6657
I20260619 14:50:13.896620 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:13.897204 964 external_mini_cluster.cc:1428] Setting key 323301cea15d1d718c87106a33fdc1d3
Jun 19 14:50:13 dist-test-slave-p3ml krb5kdc[6551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:13.907114 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:13.890453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:60895 (local address 127.0.241.62:36715)
0619 14:50:13.890885 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:13.890889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:13.891734 (+ 845us) server_negotiation.cc:407] Connection header received
0619 14:50:13.893471 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:13.893477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:13.893561 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:13.893677 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:13.895061 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:13.895847 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:13.897476 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:13.897992 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:13.898125 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:13.901748 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:13.901753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:13.901756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:13.901785 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:13.904301 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:13.904970 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:13.904977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:13.904980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:13.905096 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:13.905507 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:13.905514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:13.905517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:13.905772 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:13.905921 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:13.906545 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 14:50:13.906750 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260619 14:50:13.908068 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:13.890826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:36715 (local address 127.0.241.1:60895)
0619 14:50:13.891592 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:13.891629 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:13.893201 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:13.893895 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:13.893903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:13.894382 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:13.894903 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:13.894919 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:13.896002 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:13.896006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:13.897335 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:13.897355 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:13.898212 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:13.898220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:13.898464 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:13.899131 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:13.899152 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:13.901546 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:13.904436 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:13.904444 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:13.904458 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:13.904854 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:13.905210 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:13.905215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:13.905219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:13.905400 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:13.905894 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:13.905901 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:13.906206 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 14:50:13.906457 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 14:50:13.906938 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":277,"threads_started":1}
I20260619 14:50:13.909600 6785 heartbeater.cc:347] Connected to a master server at 127.0.241.62:36715
I20260619 14:50:13.909899 6785 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:13.910771 6785 heartbeater.cc:511] Master 127.0.241.62:36715 requested a full tablet report, sending...
I20260619 14:50:13.912848 6596 ts_manager.cc:194] Registered new tserver with Master: 3864c241055745458a2613ccad2ef715 (127.0.241.1:32927)
I20260619 14:50:13.914232 6596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:60895
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:50:13.942021 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:36715
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:14.252447 6791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:14.252712 6791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:14.252869 6791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:14.259920 6791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:14.260047 6791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:14.260233 6791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:50:14.270704 6791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:36715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:14.272063 6791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:14.273322 6791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:14.286839 6797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:14.287590 6799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:14.287618 6796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:14.289008 6791 server_base.cc:1034] running on GCE node
I20260619 14:50:14.289584 6791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:14.298669 6791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:14.299893 6791 hybrid_clock.cc:648] HybridClock initialized: now 1781880614299834 us; error 39 us; skew 500 ppm
I20260619 14:50:14.300203 6791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:14.303262 6791 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:50:14.304709 6791 webserver.cc:460] Webserver started at http://127.0.241.2:39841/ using document root <none> and password file <none>
I20260619 14:50:14.305298 6791 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:14.305392 6791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:14.305619 6791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:14.308434 6791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "ff2c4f9da28840669d52d48213ec5596"
format_stamp: "Formatted at 2026-06-19 14:50:14 on dist-test-slave-p3ml"
server_key: "03aeb18ec08479a659a71882c5c7d4d6"
server_key_iv: "18d5ed39964f6da36bb090ea5ae3886b"
server_key_version: "encryptionkey@0"
I20260619 14:50:14.309119 6791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "ff2c4f9da28840669d52d48213ec5596"
format_stamp: "Formatted at 2026-06-19 14:50:14 on dist-test-slave-p3ml"
server_key: "03aeb18ec08479a659a71882c5c7d4d6"
server_key_iv: "18d5ed39964f6da36bb090ea5ae3886b"
server_key_version: "encryptionkey@0"
I20260619 14:50:14.313781 6791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:50:14.316936 6806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:14.318127 6791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:50:14.318284 6791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "ff2c4f9da28840669d52d48213ec5596"
format_stamp: "Formatted at 2026-06-19 14:50:14 on dist-test-slave-p3ml"
server_key: "03aeb18ec08479a659a71882c5c7d4d6"
server_key_iv: "18d5ed39964f6da36bb090ea5ae3886b"
server_key_version: "encryptionkey@0"
I20260619 14:50:14.318454 6791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:14.336822 6791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:14.340765 6791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:14.340986 6791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:14.341816 6791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:14.343003 6791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:14.343067 6791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:14.343163 6791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:14.343205 6791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:14.375082 6791 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:34213
I20260619 14:50:14.375124 6918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:34213 every 8 connection(s)
I20260619 14:50:14.376380 6791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:14.379417 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 6791
I20260619 14:50:14.379635 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:14.380187 964 external_mini_cluster.cc:1428] Setting key 29849ba4eaae538c738d32a8efedfefc
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:14.395927 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:14.378871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:50979 (local address 127.0.241.62:36715)
0619 14:50:14.379060 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:14.379064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:14.380362 (+ 1298us) server_negotiation.cc:407] Connection header received
0619 14:50:14.382130 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:14.382135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:14.382227 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:14.382349 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:14.383899 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.384611 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.386155 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.386617 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.386785 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:14.390478 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:14.390483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:14.390486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:14.390514 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:14.393061 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:14.393751 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:14.393757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:14.393766 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:14.393880 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:14.394302 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:14.394307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:14.394311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:14.394566 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:14.394719 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:14.395355 (+ 636us) server_negotiation.cc:299] Negotiation successful
0619 14:50:14.395591 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:50:14.396847 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:14.379397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:36715 (local address 127.0.241.2:50979)
0619 14:50:14.380192 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:14.380227 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:14.381870 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:14.382590 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:14.382601 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:14.383091 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:14.383729 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:14.383749 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.384761 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.384768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:14.386001 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:14.386012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.386838 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.386847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:14.387080 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:14.387757 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:14.387781 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:14.390236 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:14.393198 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:14.393208 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:14.393223 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:14.393616 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:14.393988 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:14.393994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:14.393998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:14.394185 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:14.394700 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:14.394707 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:14.395017 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 14:50:14.395266 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 14:50:14.395785 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":325,"threads_started":1}
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
I20260619 14:50:14.398309 6919 heartbeater.cc:347] Connected to a master server at 127.0.241.62:36715
I20260619 14:50:14.398593 6919 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:14.399324 6919 heartbeater.cc:511] Master 127.0.241.62:36715 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
I20260619 14:50:14.400794 6596 ts_manager.cc:194] Registered new tserver with Master: ff2c4f9da28840669d52d48213ec5596 (127.0.241.2:34213)
I20260619 14:50:14.401561 6596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:50979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:50:14.422663 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:36715
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:14.734148 6924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:14.734431 6924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:14.734524 6924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:14.741927 6924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:14.742041 6924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:14.742161 6924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:50:14.751888 6924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:36715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:14.753105 6924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:14.754361 6924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:14.767131 6930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:14.767231 6929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:14.767745 6932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:14.767921 6924 server_base.cc:1034] running on GCE node
I20260619 14:50:14.768509 6924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:14.769454 6924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:14.770668 6924 hybrid_clock.cc:648] HybridClock initialized: now 1781880614770625 us; error 56 us; skew 500 ppm
I20260619 14:50:14.770972 6924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:14.773924 6924 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:50:14.775206 6924 webserver.cc:460] Webserver started at http://127.0.241.3:38237/ using document root <none> and password file <none>
I20260619 14:50:14.775920 6924 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:14.776065 6924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:14.776363 6924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:14.779202 6924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "1d2474c8e81c4a04ad041ff2633655c9"
format_stamp: "Formatted at 2026-06-19 14:50:14 on dist-test-slave-p3ml"
server_key: "c546a7cc206b2cf779830a3ab6ee31ac"
server_key_iv: "c4d303c0c4f13dcb48720bcbec243c5d"
server_key_version: "encryptionkey@0"
I20260619 14:50:14.780009 6924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "1d2474c8e81c4a04ad041ff2633655c9"
format_stamp: "Formatted at 2026-06-19 14:50:14 on dist-test-slave-p3ml"
server_key: "c546a7cc206b2cf779830a3ab6ee31ac"
server_key_iv: "c4d303c0c4f13dcb48720bcbec243c5d"
server_key_version: "encryptionkey@0"
I20260619 14:50:14.784749 6924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 14:50:14.787938 6939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:14.789115 6924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260619 14:50:14.789319 6924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "1d2474c8e81c4a04ad041ff2633655c9"
format_stamp: "Formatted at 2026-06-19 14:50:14 on dist-test-slave-p3ml"
server_key: "c546a7cc206b2cf779830a3ab6ee31ac"
server_key_iv: "c4d303c0c4f13dcb48720bcbec243c5d"
server_key_version: "encryptionkey@0"
I20260619 14:50:14.789520 6924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:14.811796 6924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:14.815798 6924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:14.816083 6924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:14.817010 6924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:14.818353 6924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:14.818454 6924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:14.818600 6924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:14.818671 6924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:14.852022 6924 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:32925
I20260619 14:50:14.852064 7051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:32925 every 8 connection(s)
I20260619 14:50:14.853235 6924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:14.860683 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 6924
I20260619 14:50:14.860899 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:14.861403 964 external_mini_cluster.cc:1428] Setting key ef6c8de60a4106dd53a920109cc41b86
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:14.872299 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:14.855687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:43075 (local address 127.0.241.62:36715)
0619 14:50:14.855852 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:14.855857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:14.856950 (+ 1093us) server_negotiation.cc:407] Connection header received
0619 14:50:14.858663 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:14.858668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:14.858730 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:14.858846 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:14.860390 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.861534 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.863385 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.864020 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.864194 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:14.867357 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:14.867361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:14.867364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:14.867393 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:14.869703 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:14.870341 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:14.870345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:14.870347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:14.870455 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:14.870832 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:14.870835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:14.870837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:14.871069 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:14.871171 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:14.871906 (+ 735us) server_negotiation.cc:299] Negotiation successful
0619 14:50:14.872053 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:50:14.873234 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:14.856087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:36715 (local address 127.0.241.3:43075)
0619 14:50:14.856787 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:14.856828 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:14.858410 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:14.859048 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:14.859056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:14.859657 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:14.860212 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:14.860228 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.861694 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.861698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:14.863230 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:14.863244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.864259 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.864265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:14.864514 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:14.865154 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:14.865174 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:14.867190 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:14.869823 (+ 2633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:14.869829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:14.869841 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:14.870245 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:14.870539 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:14.870542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:14.870544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:14.870745 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:14.871169 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:14.871174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:14.871548 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 14:50:14.871824 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 14:50:14.872178 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":274,"threads_started":1}
I20260619 14:50:14.874488 7052 heartbeater.cc:347] Connected to a master server at 127.0.241.62:36715
I20260619 14:50:14.874804 7052 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:14.875690 7052 heartbeater.cc:511] Master 127.0.241.62:36715 requested a full tablet report, sending...
I20260619 14:50:14.877110 6596 ts_manager.cc:194] Registered new tserver with Master: 1d2474c8e81c4a04ad041ff2633655c9 (127.0.241.3:32925)
I20260619 14:50:14.877882 6596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:43075
I20260619 14:50:14.888895 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/test-admin.keytab.
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:14.916625 964 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
I20260619 14:50:14.917344 6785 heartbeater.cc:503] Master 127.0.241.62:36715 was elected leader, sending a full tablet report...
W20260619 14:50:14.922621 7059 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:14.919670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.0.241.1:32927)
0619 14:50:14.920895 (+ 1225us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:14.920902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:14.920924 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:50:14.920971 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:14.920976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:14.921237 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:14.921363 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:14.922215 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35800: BlockingRecv error: recv got EOF from 127.0.0.1:35800 (error 108)
Metrics: {"server-negotiator.queue_time_us":1081,"thread_start_us":920,"threads_started":1}
W20260619 14:50:14.922997 964 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:14.924527 964 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:14.925890 964 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:50:14.940955 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:14.927501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.0.241.1:32927)
0619 14:50:14.927697 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:14.927704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:14.928000 (+ 296us) server_negotiation.cc:407] Connection header received
0619 14:50:14.928259 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:14.928263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:14.928319 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:14.928397 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:14.929341 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.930337 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.931560 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.932100 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.932200 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:14.935590 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:14.935601 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:14.935615 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:14.935645 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:14.938395 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:14.939052 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:14.939058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:14.939060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:14.939165 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:14.939542 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:14.939545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:14.939547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:14.939846 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:14.940072 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:14.940407 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 14:50:14.940678 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 14:50:14.946076 7059 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:14.944618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35818 (local address 127.0.241.1:32927)
0619 14:50:14.944841 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:14.944847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:14.944997 (+ 150us) server_negotiation.cc:407] Connection header received
0619 14:50:14.945114 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:14.945117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:14.945177 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:14.945250 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:14.945851 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35818: BlockingRecv error: recv got EOF from 127.0.0.1:35818 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 14:50:14.946179 964 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:14.947818 964 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:14.949169 964 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:50:14.962492 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:14.950458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.0.241.1:32927)
0619 14:50:14.950630 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:14.950636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:14.950853 (+ 217us) server_negotiation.cc:407] Connection header received
0619 14:50:14.950976 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:14.950980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:14.951025 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:14.951090 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:14.952055 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.952707 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.953907 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.954313 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.954395 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:14.957729 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:14.957733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:14.957736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:14.957775 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:14.960232 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:14.960811 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:14.960835 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:14.960840 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:14.960990 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:14.961361 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:14.961366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:14.961368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:14.961660 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:14.961843 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:14.962055 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 14:50:14.962187 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 14:50:14.966511 7059 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:14.965172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.0.241.1:32927)
0619 14:50:14.965346 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:14.965350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:14.965449 (+ 99us) server_negotiation.cc:407] Connection header received
0619 14:50:14.965559 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:14.965562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:14.965619 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:14.965691 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:14.966317 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35834: BlockingRecv error: recv got EOF from 127.0.0.1:35834 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 14:50:14.966989 964 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880565277815-964-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:14.968572 964 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:14.969954 964 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:50:14 dist-test-slave-p3ml krb5kdc[6551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:50:14.983557 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:14.971102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35838 (local address 127.0.241.1:32927)
0619 14:50:14.971277 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:14.971282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:14.971567 (+ 285us) server_negotiation.cc:407] Connection header received
0619 14:50:14.971690 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:14.971693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:14.971739 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:14.971816 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:14.972753 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.973409 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.974634 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.975030 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.975116 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:14.978409 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:14.978413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:14.978416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:14.978442 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:14.981196 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:14.981872 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:14.981875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:14.981877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:14.982006 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:14.982333 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:14.982336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:14.982338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:14.982543 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:14.982750 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:14.982986 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 14:50:14.983126 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:50:14.985270 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 6657
I20260619 14:50:14.997074 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:14.985935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.0.241.1:32927)
0619 14:50:14.986065 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:14.986069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:14.986470 (+ 401us) server_negotiation.cc:407] Connection header received
0619 14:50:14.986578 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:14.986581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:14.986628 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:14.986700 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:14.987528 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.988150 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.989364 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:14.989980 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:14.990095 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:14.992053 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:14.992058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:14.992061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:14.992090 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:14.994803 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:14.995401 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:14.995405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:14.995407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:14.995632 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:14.995971 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:14.995975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:14.995977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:14.996208 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:14.996408 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:14.996651 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 14:50:14.996811 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:50:14.997627 6780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:15.080025 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 6657
I20260619 14:50:15.096941 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 6791
Jun 19 14:50:15 dist-test-slave-p3ml krb5kdc[6551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:50:15.111747 7072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:15.097805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.0.241.2:34213)
0619 14:50:15.098319 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:15.098326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:15.098349 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:50:15.098406 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:15.098413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:15.098639 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:15.098808 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:15.099861 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:15.100845 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:15.102040 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:15.102597 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:15.102734 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:15.106654 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:15.106665 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:15.106679 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:15.106710 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:15.109397 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:15.109999 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:15.110005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:15.110007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:15.110128 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:15.110441 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:15.110444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:15.110446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:15.110743 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:15.110962 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:15.111207 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 14:50:15.111399 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":255,"threads_started":1}
I20260619 14:50:15.112946 6914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:15.194208 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 6791
I20260619 14:50:15.210152 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 6924
Jun 19 14:50:15 dist-test-slave-p3ml krb5kdc[6551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:50:15.226348 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:15.211519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.0.241.3:32925)
0619 14:50:15.212181 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:15.212187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:15.212216 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:50:15.212271 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:15.212277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:15.212491 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:15.212629 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:15.213817 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:15.214835 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:15.216224 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:15.216790 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:15.216889 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:15.220832 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:15.220843 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:15.220858 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:15.220892 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:15.223892 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:15.224564 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:15.224571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:15.224574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:15.224684 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:15.225003 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:15.225006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:15.225008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:15.225320 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:15.225555 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:15.225759 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 14:50:15.226020 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":365,"threads_started":1}
I20260619 14:50:15.227846 7047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:15.311880 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 6924
I20260619 14:50:15.327839 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 6564
I20260619 14:50:15.329216 6626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:15.430877 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 6564
2026-06-19T14:50:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3842 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:50:15 dist-test-slave-p3ml krb5kdc[7079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:50:15 dist-test-slave-p3ml krb5kdc[7079](info): set up 2 sockets
Jun 19 14:50:15 dist-test-slave-p3ml krb5kdc[7079](info): commencing operation
krb5kdc: starting...
W20260619 14:50:16.536309 964 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:50:16 dist-test-slave-p3ml krb5kdc[7079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.62.keytab.
Entry for principal oryx/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.62.keytab.
I20260619 14:50:16.746410 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33289
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.62.keytab
--principal=oryx/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:33289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:17.060246 7092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:17.060529 7092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:17.060698 7092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:17.067821 7092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:17.067917 7092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:17.068018 7092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:17.068114 7092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:17.068197 7092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:17.077695 7092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:33289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.62.keytab
--principal=oryx/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:17.078954 7092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:17.080271 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:17.091982 7098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:17.092090 7097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:17.092115 7100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:17.092762 7092 server_base.cc:1034] running on GCE node
I20260619 14:50:17.093556 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:17.094946 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:17.096153 7092 hybrid_clock.cc:648] HybridClock initialized: now 1781880617096103 us; error 44 us; skew 500 ppm
I20260619 14:50:17.096441 7092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:17 dist-test-slave-p3ml krb5kdc[7079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:17.099339 7092 init.cc:377] Logged in from keytab as oryx/127.0.241.62@KRBTEST.COM (short username oryx)
I20260619 14:50:17.100744 7092 webserver.cc:460] Webserver started at http://127.0.241.62:42699/ using document root <none> and password file <none>
I20260619 14:50:17.101411 7092 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:17.101555 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:17.101851 7092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:17.104729 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "4002d8e4b9404f0c9d8901c3203d3483"
format_stamp: "Formatted at 2026-06-19 14:50:17 on dist-test-slave-p3ml"
server_key: "0d81c6673ec74484b9e40771cbce3ff1"
server_key_iv: "aca6003ace7ccb6f037c6447a889dad4"
server_key_version: "encryptionkey@0"
I20260619 14:50:17.105466 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "4002d8e4b9404f0c9d8901c3203d3483"
format_stamp: "Formatted at 2026-06-19 14:50:17 on dist-test-slave-p3ml"
server_key: "0d81c6673ec74484b9e40771cbce3ff1"
server_key_iv: "aca6003ace7ccb6f037c6447a889dad4"
server_key_version: "encryptionkey@0"
I20260619 14:50:17.110272 7092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 14:50:17.113549 7107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:17.114784 7092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:50:17.114950 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "4002d8e4b9404f0c9d8901c3203d3483"
format_stamp: "Formatted at 2026-06-19 14:50:17 on dist-test-slave-p3ml"
server_key: "0d81c6673ec74484b9e40771cbce3ff1"
server_key_iv: "aca6003ace7ccb6f037c6447a889dad4"
server_key_version: "encryptionkey@0"
I20260619 14:50:17.115128 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:17.128306 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:17.131963 7092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:17.132238 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:17.149922 7092 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:33289
I20260619 14:50:17.149956 7158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:33289 every 8 connection(s)
I20260619 14:50:17.151247 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:17.154752 7159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:17.155946 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 7092
I20260619 14:50:17.156211 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:17.157016 964 external_mini_cluster.cc:1428] Setting key 27abec4d14ed6eae93ce2d5be1e415db
I20260619 14:50:17.167353 7159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483: Bootstrap starting.
Jun 19 14:50:17 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:17.170532 7159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:17.171593 7159 log.cc:824] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:17.174325 7159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483: No bootstrap required, opened a new log
I20260619 14:50:17.178040 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:17.159110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.0.241.62:33289)
0619 14:50:17.159978 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:17.159990 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:17.160032 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:50:17.161545 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:17.161566 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:17.161937 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:17.162341 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:17.163822 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:17.165122 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:17.166317 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:17.166950 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:17.167141 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:17.171114 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:17.171122 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:17.171135 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:17.171169 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:17.174213 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:17.174906 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:17.174914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:17.174928 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:17.175077 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:17.175457 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:17.175462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:17.175466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:17.175990 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:17.176319 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:17.176706 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 14:50:17.177106 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":421,"threads_started":1}
I20260619 14:50:17.182235 7159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } }
I20260619 14:50:17.182592 7159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:17.182699 7159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4002d8e4b9404f0c9d8901c3203d3483, State: Initialized, Role: FOLLOWER
I20260619 14:50:17.183286 7159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } }
I20260619 14:50:17.183573 7159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:17.183737 7159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:17.183924 7159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:17.185976 7159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } }
I20260619 14:50:17.186444 7159 leader_election.cc:302] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4002d8e4b9404f0c9d8901c3203d3483; no voters:
I20260619 14:50:17.187099 7159 leader_election.cc:288] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:17.187247 7164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:17.187995 7164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 1 LEADER]: Becoming Leader. State: Replica: 4002d8e4b9404f0c9d8901c3203d3483, State: Running, Role: LEADER
I20260619 14:50:17.188545 7164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } }
I20260619 14:50:17.189178 7159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:17.190943 7166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4002d8e4b9404f0c9d8901c3203d3483. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } }
I20260619 14:50:17.191242 7166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:17.191686 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } }
I20260619 14:50:17.192128 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:17.193902 7170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:17.200148 7170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:17.210844 7170 catalog_manager.cc:1347] Generated new cluster ID: d1e6b37ed3e14281bc735000d4ab54a6
I20260619 14:50:17.210956 7170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:17.226125 7170 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:17.227952 7170 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:17.238588 7170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483: Generated new TSK 0
I20260619 14:50:17.239645 7170 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab.
Entry for principal oryx/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab.
I20260619 14:50:17.288986 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33289
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab
--principal=oryx/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:17.614281 7185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:17.614547 7185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:17.614745 7185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:17.622169 7185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:17.622265 7185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:17.622463 7185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:50:17.631983 7185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab
--principal=oryx/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:17.633246 7185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:17.634539 7185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:17.647732 7190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:17.647653 7191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:17.647692 7193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:17.648654 7185 server_base.cc:1034] running on GCE node
I20260619 14:50:17.649257 7185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:17.650192 7185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:17.651407 7185 hybrid_clock.cc:648] HybridClock initialized: now 1781880617651350 us; error 59 us; skew 500 ppm
I20260619 14:50:17.651759 7185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:17 dist-test-slave-p3ml krb5kdc[7079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:17.654837 7185 init.cc:377] Logged in from keytab as oryx/127.0.241.1@KRBTEST.COM (short username oryx)
I20260619 14:50:17.656320 7185 webserver.cc:460] Webserver started at http://127.0.241.1:45319/ using document root <none> and password file <none>
I20260619 14:50:17.656937 7185 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:17.657045 7185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:17.657276 7185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:17.660148 7185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "6b840e7d2eb14f1e9cc4094eecbe287b"
format_stamp: "Formatted at 2026-06-19 14:50:17 on dist-test-slave-p3ml"
server_key: "0bbf4cf58301f2616a6ce70628f0fc02"
server_key_iv: "932536dd458e1c7e682318eebf2788d0"
server_key_version: "encryptionkey@0"
I20260619 14:50:17.660861 7185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "6b840e7d2eb14f1e9cc4094eecbe287b"
format_stamp: "Formatted at 2026-06-19 14:50:17 on dist-test-slave-p3ml"
server_key: "0bbf4cf58301f2616a6ce70628f0fc02"
server_key_iv: "932536dd458e1c7e682318eebf2788d0"
server_key_version: "encryptionkey@0"
I20260619 14:50:17.665640 7185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:50:17.668949 7200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:17.670172 7185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:50:17.670324 7185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "6b840e7d2eb14f1e9cc4094eecbe287b"
format_stamp: "Formatted at 2026-06-19 14:50:17 on dist-test-slave-p3ml"
server_key: "0bbf4cf58301f2616a6ce70628f0fc02"
server_key_iv: "932536dd458e1c7e682318eebf2788d0"
server_key_version: "encryptionkey@0"
I20260619 14:50:17.670512 7185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:17.685279 7185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:17.689271 7185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:17.689554 7185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:17.690476 7185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:17.691741 7185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:17.691843 7185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:17.691932 7185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:17.691978 7185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:17.726485 7185 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:38131
I20260619 14:50:17.726528 7312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:38131 every 8 connection(s)
I20260619 14:50:17.727818 7185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:17.729460 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 7185
I20260619 14:50:17.729673 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:17.730491 964 external_mini_cluster.cc:1428] Setting key 219566dfa92bd84b4046cd2c02dad628
Jun 19 14:50:17 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:17.749578 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:17.730590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:57167 (local address 127.0.241.62:33289)
0619 14:50:17.731104 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:17.731112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:17.732264 (+ 1152us) server_negotiation.cc:407] Connection header received
0619 14:50:17.734041 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:17.734050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:17.734139 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:17.734260 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:17.735894 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:17.737107 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:17.739021 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:17.739398 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:17.739565 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:17.743707 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:17.743712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:17.743716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:17.743746 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:17.746069 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:17.746800 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:17.746810 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:17.746814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:17.746941 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:17.747498 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:17.747506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:17.747513 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:17.747910 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:17.748067 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:17.748796 (+ 729us) server_negotiation.cc:299] Negotiation successful
0619 14:50:17.749084 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":248,"threads_started":1}
I20260619 14:50:17.750134 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:17.731178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.1:57167)
0619 14:50:17.732069 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:17.732145 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:17.733780 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:17.734465 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:17.734474 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:17.735001 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:17.735723 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:17.735744 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:17.737269 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:17.737273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:17.738862 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:17.738879 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:17.739678 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:17.739688 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:17.740010 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:17.740711 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:17.740744 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:17.743490 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:17.746218 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:17.746227 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:17.746245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:17.746668 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:17.747052 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:17.747057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:17.747061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:17.747302 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:17.748045 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:17.748051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:17.748401 (+ 350us) client_negotiation.cc:769] Sending connection context
0619 14:50:17.748624 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 14:50:17.749022 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":412,"threads_started":1}
WARNING: no policy specified for oryx/127.0.241.2@KRBTEST.COM; defaulting to no policy
I20260619 14:50:17.751718 7313 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33289
I20260619 14:50:17.752033 7313 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:17.752801 7313 heartbeater.cc:511] Master 127.0.241.62:33289 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.2@KRBTEST.COM" created.
I20260619 14:50:17.755019 7124 ts_manager.cc:194] Registered new tserver with Master: 6b840e7d2eb14f1e9cc4094eecbe287b (127.0.241.1:38131)
I20260619 14:50:17.756484 7124 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:57167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab.
Entry for principal oryx/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab.
I20260619 14:50:17.778868 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33289
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab
--principal=oryx/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:18.095185 7319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:18.095495 7319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:18.095613 7319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:18.103047 7319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:18.103143 7319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:18.103256 7319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:50:18.112659 7319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab
--principal=oryx/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:18.113808 7319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:18.115042 7319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:18.131738 7327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:18.131788 7325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:18.131834 7324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:18.132723 7319 server_base.cc:1034] running on GCE node
I20260619 14:50:18.133337 7319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:18.134214 7319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:18.135403 7319 hybrid_clock.cc:648] HybridClock initialized: now 1781880618135352 us; error 41 us; skew 500 ppm
I20260619 14:50:18.135778 7319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:18 dist-test-slave-p3ml krb5kdc[7079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:18.138955 7319 init.cc:377] Logged in from keytab as oryx/127.0.241.2@KRBTEST.COM (short username oryx)
I20260619 14:50:18.140359 7319 webserver.cc:460] Webserver started at http://127.0.241.2:43339/ using document root <none> and password file <none>
I20260619 14:50:18.141010 7319 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:18.141137 7319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:18.141425 7319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:18.144289 7319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "a6e46bb4f46e4d309506a4167f43343f"
format_stamp: "Formatted at 2026-06-19 14:50:18 on dist-test-slave-p3ml"
server_key: "72d117364a365176047c5b837507efe8"
server_key_iv: "a7252824d4f6d6b667b1ecb31b1abe18"
server_key_version: "encryptionkey@0"
I20260619 14:50:18.145037 7319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "a6e46bb4f46e4d309506a4167f43343f"
format_stamp: "Formatted at 2026-06-19 14:50:18 on dist-test-slave-p3ml"
server_key: "72d117364a365176047c5b837507efe8"
server_key_iv: "a7252824d4f6d6b667b1ecb31b1abe18"
server_key_version: "encryptionkey@0"
I20260619 14:50:18.150061 7319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:50:18.154192 7334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:18.155697 7319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:50:18.155843 7319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "a6e46bb4f46e4d309506a4167f43343f"
format_stamp: "Formatted at 2026-06-19 14:50:18 on dist-test-slave-p3ml"
server_key: "72d117364a365176047c5b837507efe8"
server_key_iv: "a7252824d4f6d6b667b1ecb31b1abe18"
server_key_version: "encryptionkey@0"
I20260619 14:50:18.156031 7319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:18.181223 7319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:18.185693 7319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:18.185985 7319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:18.186865 7319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:18.188254 7319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:18.188333 7319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:18.188421 7319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:18.188470 7319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:18.222576 7319 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:38485
I20260619 14:50:18.222615 7446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:38485 every 8 connection(s)
I20260619 14:50:18.223881 7319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:18.228168 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 7319
I20260619 14:50:18.228389 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:18.228940 964 external_mini_cluster.cc:1428] Setting key 58fb3d1c601c7b5c2e5671a95f2dc5c2
Jun 19 14:50:18 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:18.246881 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:18.226402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:54493 (local address 127.0.241.62:33289)
0619 14:50:18.226628 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:18.226635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:18.227804 (+ 1169us) server_negotiation.cc:407] Connection header received
0619 14:50:18.230176 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:18.230183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:18.230261 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:18.230373 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:18.232053 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:18.232844 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:18.235237 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:18.235711 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:18.235811 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:18.240095 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:18.240104 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:18.240112 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:18.240154 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:18.243781 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:18.244484 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:18.244492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:18.244504 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:18.244674 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:18.245050 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:18.245058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:18.245064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:18.245413 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:18.245565 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:18.246176 (+ 611us) server_negotiation.cc:299] Negotiation successful
0619 14:50:18.246421 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 14:50:18.247502 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:18.226881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.2:54493)
0619 14:50:18.227631 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:18.227673 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:18.229856 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:18.230568 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:18.230576 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:18.231044 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:18.231866 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:18.231894 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:18.233010 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:18.233017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:18.235083 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:18.235099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:18.235934 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:18.235944 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:18.236256 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:18.237064 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:18.237096 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:18.239884 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:18.243917 (+ 4033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:18.243923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:18.243934 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:18.244334 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:18.244778 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:18.244781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:18.244783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:18.244938 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:18.245534 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:18.245539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:18.245824 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 14:50:18.246035 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 14:50:18.246397 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":287,"threads_started":1}
I20260619 14:50:18.249035 7447 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33289
I20260619 14:50:18.249413 7447 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:18.250273 7447 heartbeater.cc:511] Master 127.0.241.62:33289 requested a full tablet report, sending...
I20260619 14:50:18.251953 7124 ts_manager.cc:194] Registered new tserver with Master: a6e46bb4f46e4d309506a4167f43343f (127.0.241.2:38485)
WARNING: no policy specified for oryx/127.0.241.3@KRBTEST.COM; defaulting to no policy
I20260619 14:50:18.252868 7124 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:54493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab.
Entry for principal oryx/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab.
I20260619 14:50:18.279096 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33289
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab
--principal=oryx/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:18.598596 7452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:18.598937 7452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:18.599086 7452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:18.606422 7452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:18.606594 7452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:18.606777 7452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:50:18.616405 7452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab
--principal=oryx/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:18.617632 7452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:18.619067 7452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:18.631906 7457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:18.631856 7458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:18.632095 7460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:18.633085 7452 server_base.cc:1034] running on GCE node
I20260619 14:50:18.633740 7452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:18.634657 7452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:18.635881 7452 hybrid_clock.cc:648] HybridClock initialized: now 1781880618635821 us; error 48 us; skew 500 ppm
I20260619 14:50:18.636204 7452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:18 dist-test-slave-p3ml krb5kdc[7079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:18.639302 7452 init.cc:377] Logged in from keytab as oryx/127.0.241.3@KRBTEST.COM (short username oryx)
I20260619 14:50:18.640668 7452 webserver.cc:460] Webserver started at http://127.0.241.3:41279/ using document root <none> and password file <none>
I20260619 14:50:18.641296 7452 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:18.641392 7452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:18.641629 7452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:18.644450 7452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "8cac0c45467f435a83c18d10cc73685a"
format_stamp: "Formatted at 2026-06-19 14:50:18 on dist-test-slave-p3ml"
server_key: "82a2c7eba532c94b22d00d8c031600ff"
server_key_iv: "e213d2489b1d7f02d12bde6c24ebb933"
server_key_version: "encryptionkey@0"
I20260619 14:50:18.645166 7452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "8cac0c45467f435a83c18d10cc73685a"
format_stamp: "Formatted at 2026-06-19 14:50:18 on dist-test-slave-p3ml"
server_key: "82a2c7eba532c94b22d00d8c031600ff"
server_key_iv: "e213d2489b1d7f02d12bde6c24ebb933"
server_key_version: "encryptionkey@0"
I20260619 14:50:18.649926 7452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:50:18.653262 7467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:18.654564 7452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:50:18.654767 7452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "8cac0c45467f435a83c18d10cc73685a"
format_stamp: "Formatted at 2026-06-19 14:50:18 on dist-test-slave-p3ml"
server_key: "82a2c7eba532c94b22d00d8c031600ff"
server_key_iv: "e213d2489b1d7f02d12bde6c24ebb933"
server_key_version: "encryptionkey@0"
I20260619 14:50:18.654958 7452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:18.690768 7452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:18.695048 7452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:18.695353 7452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:18.696440 7452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:18.697880 7452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:18.697952 7452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:18.698129 7452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:18.698179 7452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:18.731019 7452 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:44993
I20260619 14:50:18.731037 7579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:44993 every 8 connection(s)
I20260619 14:50:18.732564 7452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:18.738065 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 7452
I20260619 14:50:18.738245 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:18.738795 964 external_mini_cluster.cc:1428] Setting key a888edc18f18e36108fa27a6293c2ad5
Jun 19 14:50:18 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:18.752235 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:18.735112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:41149 (local address 127.0.241.62:33289)
0619 14:50:18.735282 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:18.735286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:18.736466 (+ 1180us) server_negotiation.cc:407] Connection header received
0619 14:50:18.738358 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:18.738365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:18.738467 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:18.738583 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:18.740219 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:18.741280 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:18.742845 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:18.743256 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:18.743356 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:18.746612 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:18.746618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:18.746621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:18.746653 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:18.749492 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:18.750173 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:18.750178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:18.750180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:18.750301 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:18.750663 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:18.750667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:18.750668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:18.750936 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:18.751058 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:18.751752 (+ 694us) server_negotiation.cc:299] Negotiation successful
0619 14:50:18.751903 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:50:18.753060 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:18.735599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.3:41149)
0619 14:50:18.736299 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:18.736342 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:18.738058 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:18.738766 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:18.738774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:18.739287 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:18.740048 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:18.740077 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:18.741415 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:18.741419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:18.742719 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:18.742730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:18.743495 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:18.743501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:18.743685 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:18.744367 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:18.744392 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:18.746417 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:18.749621 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:18.749628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:18.749640 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:18.750035 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:18.750398 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:18.750401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:18.750403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:18.750560 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:18.751040 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:18.751045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:18.751330 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 14:50:18.751624 (+ 294us) client_negotiation.cc:240] Negotiation successful
0619 14:50:18.751980 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":279,"threads_started":1}
I20260619 14:50:18.754592 7580 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33289
I20260619 14:50:18.754892 7580 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:18.755631 7580 heartbeater.cc:511] Master 127.0.241.62:33289 requested a full tablet report, sending...
I20260619 14:50:18.756985 7124 ts_manager.cc:194] Registered new tserver with Master: 8cac0c45467f435a83c18d10cc73685a (127.0.241.3:44993)
I20260619 14:50:18.757808 7124 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:41149
I20260619 14:50:18.759246 7313 heartbeater.cc:503] Master 127.0.241.62:33289 was elected leader, sending a full tablet report...
I20260619 14:50:18.766016 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.241.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.61.keytab.
Entry for principal oryx/127.0.241.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.61.keytab.
I20260619 14:50:18.811363 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.61:40459
--webserver_interface=127.0.241.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.61.keytab
--principal=oryx/127.0.241.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:33289,127.0.241.61:40459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:19.125567 7585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:19.125906 7585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:19.126055 7585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:19.133076 7585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:19.133217 7585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:19.133316 7585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:19.133416 7585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:19.133512 7585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:19.143241 7585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:33289,127.0.241.61:40459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.61.keytab
--principal=oryx/127.0.241.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.61:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.241.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
W20260619 14:50:19.143641 7585 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.241.62:33289,127.0.241.61:40459'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 14:50:19.144651 7585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:19.145929 7585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:19.158257 7593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:19.158381 7591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:19.158576 7590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:19.159086 7585 server_base.cc:1034] running on GCE node
I20260619 14:50:19.159924 7585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:19.161309 7585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:19.162544 7585 hybrid_clock.cc:648] HybridClock initialized: now 1781880619162474 us; error 59 us; skew 500 ppm
I20260619 14:50:19.162863 7585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:19 dist-test-slave-p3ml krb5kdc[7079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:19.170397 7585 init.cc:377] Logged in from keytab as oryx/127.0.241.61@KRBTEST.COM (short username oryx)
I20260619 14:50:19.171962 7585 webserver.cc:460] Webserver started at http://127.0.241.61:41113/ using document root <none> and password file <none>
I20260619 14:50:19.172668 7585 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:19.172796 7585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:19.173085 7585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:19.175966 7585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data/instance:
uuid: "04fac3b86b9f46cda4436820eb3e55fb"
format_stamp: "Formatted at 2026-06-19 14:50:19 on dist-test-slave-p3ml"
server_key: "632b3185363f58531f38cc38d2f3c3dc"
server_key_iv: "c358348c8ba6249e26b97ae29b6df882"
server_key_version: "encryptionkey@0"
I20260619 14:50:19.176702 7585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/wal/instance:
uuid: "04fac3b86b9f46cda4436820eb3e55fb"
format_stamp: "Formatted at 2026-06-19 14:50:19 on dist-test-slave-p3ml"
server_key: "632b3185363f58531f38cc38d2f3c3dc"
server_key_iv: "c358348c8ba6249e26b97ae29b6df882"
server_key_version: "encryptionkey@0"
I20260619 14:50:19.181576 7585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 14:50:19.184839 7600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:19.186441 7585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:50:19.186616 7585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/wal
uuid: "04fac3b86b9f46cda4436820eb3e55fb"
format_stamp: "Formatted at 2026-06-19 14:50:19 on dist-test-slave-p3ml"
server_key: "632b3185363f58531f38cc38d2f3c3dc"
server_key_iv: "c358348c8ba6249e26b97ae29b6df882"
server_key_version: "encryptionkey@0"
I20260619 14:50:19.186833 7585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:19.209491 7585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:19.213546 7585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:19.213809 7585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:19.232762 7585 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.61:40459
I20260619 14:50:19.232781 7651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.61:40459 every 8 connection(s)
I20260619 14:50:19.233958 7585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
I20260619 14:50:19.237507 7652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:19.239631 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 7585
I20260619 14:50:19.244828 7652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:50:19 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.241.61@KRBTEST.COM
Jun 19 14:50:19 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.61@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:19.255744 7447 heartbeater.cc:503] Master 127.0.241.62:33289 was elected leader, sending a full tablet report...
I20260619 14:50:19.262351 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.247733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.0.241.62:33289)
0619 14:50:19.247922 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.247929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.248470 (+ 541us) server_negotiation.cc:407] Connection header received
0619 14:50:19.248676 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.248681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.248742 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.248866 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.249890 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.250562 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.252152 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.252595 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.252735 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.256318 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.256324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.256328 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.256364 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.259514 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.260294 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.260300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.260303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.260443 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.260852 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.260859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.260863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.261284 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:19.261431 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:19.261849 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 14:50:19.262035 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:50:19.262943 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.247684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:56486)
0619 14:50:19.248320 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.248353 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.248541 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.248919 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.248926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.249510 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:19.249743 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.249751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.250708 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.250714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.252003 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.252014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.252827 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.252834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.253001 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.253637 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:19.253671 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:19.256104 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:19.259663 (+ 3559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.259672 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.259679 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.260193 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.260602 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.260607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.260611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.260735 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.261414 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:19.261419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:19.261546 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 14:50:19.261750 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 14:50:19.261959 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":294,"threads_started":1}
I20260619 14:50:19.262979 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.241704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.0.241.61:40459)
0619 14:50:19.242525 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.242539 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.242577 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:50:19.244221 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.244242 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.244682 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.245067 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.246829 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.248743 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.249967 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.250559 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.250775 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.254753 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.254763 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.254776 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.254819 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.257992 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.258783 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.258791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.258799 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.258952 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.259396 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.259403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.259409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.260122 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:19.260395 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:19.261449 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0619 14:50:19.261808 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":409,"threads_started":1}
I20260619 14:50:19.264782 7652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:50:19 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.61@KRBTEST.COM for oryx/127.0.241.61@KRBTEST.COM
I20260619 14:50:19.276592 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.265728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:40459 (local address 127.0.0.1:42672)
0619 14:50:19.265901 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.265920 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.266040 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.266316 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.266320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.266651 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:19.266762 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.266769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.267730 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.267735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.268541 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.268550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.269188 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.269192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.269372 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.269851 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:19.269865 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:19.271797 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:19.274386 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.274389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.274391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.274843 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.275177 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.275181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.275183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.275310 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.275837 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:19.275841 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:19.275973 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 14:50:19.276092 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 14:50:19.276297 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 14:50:19.276710 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.265719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.0.241.61:40459)
0619 14:50:19.265888 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.265893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.265990 (+ 97us) server_negotiation.cc:407] Connection header received
0619 14:50:19.266164 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.266167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.266212 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.266303 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.266906 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.267604 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.268677 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.269057 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.269183 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.271945 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.271950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.271953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.271979 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.274274 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.274938 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.274942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.274944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.275044 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.275399 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.275402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.275404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.275708 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:19.275813 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:19.276159 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 14:50:19.276422 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":50}
I20260619 14:50:19.282546 7652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: Bootstrap starting.
I20260619 14:50:19.285750 7652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:19.286710 7652 log.cc:824] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:19.289242 7652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: No bootstrap required, opened a new log
I20260619 14:50:19.296227 7652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } }
I20260619 14:50:19.296538 7652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:19.296648 7652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fac3b86b9f46cda4436820eb3e55fb, State: Initialized, Role: FOLLOWER
I20260619 14:50:19.297256 7652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } }
I20260619 14:50:19.298046 7658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } } }
I20260619 14:50:19.298323 7658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:50:19.299001 7652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:50:19.307624 7669 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:50:19.307796 7669 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:50:19.309178 7585 master_runner.cc:428] Detected that this master 04fac3b86b9f46cda4436820eb3e55fb is joining an existing cluster
I20260619 14:50:19.309285 7585 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.241.61:40459
I20260619 14:50:19.329649 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.312927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.0.241.62:33289)
0619 14:50:19.313119 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.313125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.314349 (+ 1224us) server_negotiation.cc:407] Connection header received
0619 14:50:19.314575 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.314581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.314680 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.314751 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.316282 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.317403 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.319568 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.320217 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.320317 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.322829 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.322836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.322840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.322879 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.326326 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.327228 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.327233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.327238 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.327385 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.327869 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.327876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.327881 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.328236 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:19.328374 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:19.329070 (+ 696us) server_negotiation.cc:299] Negotiation successful
0619 14:50:19.329284 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:50:19.330699 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.313529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42678 (local address 127.0.241.61:40459)
0619 14:50:19.313735 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.313743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.314159 (+ 416us) server_negotiation.cc:407] Connection header received
0619 14:50:19.314370 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.314375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.314433 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.314534 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.315673 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.316793 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.318903 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.319848 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.319976 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.322755 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.322762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.322770 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.322806 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.327131 (+ 4325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.327946 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.327953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.327959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.328075 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.328530 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.328538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.328544 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.328954 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:19.329104 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:19.329557 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 14:50:19.330120 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":36,"spinlock_wait_cycles":37632}
I20260619 14:50:19.330802 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.313170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:56496)
0619 14:50:19.314254 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.314272 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.314438 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.314947 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.314952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.315705 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:19.316097 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.316109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.317613 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.317618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.319383 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.319397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.320595 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.320601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.320821 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.321397 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:19.321418 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:19.322604 (+ 1186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:19.326469 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.326474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.326478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.327095 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.327554 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.327558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.327562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.327735 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.328482 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:19.328488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:19.328742 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 14:50:19.329832 (+ 1090us) client_negotiation.cc:240] Negotiation successful
0619 14:50:19.330316 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":217,"thread_start_us":771,"threads_started":1}
I20260619 14:50:19.333004 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.313142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:40459 (local address 127.0.0.1:42678)
0619 14:50:19.314059 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.314081 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.314250 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.314546 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.314550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.315062 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:19.315425 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.315554 (+ 129us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.316996 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.317001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.318741 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.318759 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.320245 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.320253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.320469 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.321185 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:19.321212 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:19.322549 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:19.327267 (+ 4718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.327274 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.327281 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.327830 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.328199 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.328225 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.328232 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.328404 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.329158 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:19.329165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:19.329311 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 14:50:19.331345 (+ 2034us) client_negotiation.cc:240] Negotiation successful
0619 14:50:19.332437 (+ 1092us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":421,"threads_started":1}
WARNING: no policy specified for oryx/127.0.241.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.60@KRBTEST.COM" created.
Jun 19 14:50:19 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.62@KRBTEST.COM for oryx/127.0.241.61@KRBTEST.COM
I20260619 14:50:19.354552 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.337600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:40459 (local address 127.0.0.1:42690)
0619 14:50:19.339960 (+ 2360us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.340004 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.340230 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.340776 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.340784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.341657 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:19.341864 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.341873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.343267 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.343271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.344678 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.344689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.345466 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.345471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.345654 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.346262 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:19.346283 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:19.348323 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:19.351527 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.351538 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.351546 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.352084 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.352530 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.352548 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.352551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.352717 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.353363 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:19.353371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:19.353591 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 14:50:19.353799 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 14:50:19.354119 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2124,"thread_start_us":1956,"threads_started":1}
I20260619 14:50:19.354804 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.337709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.0.241.61:40459)
0619 14:50:19.337904 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.337913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.340091 (+ 2178us) server_negotiation.cc:407] Connection header received
0619 14:50:19.340463 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.340469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.340571 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.340684 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.341999 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.343126 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.344817 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.345295 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.345440 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.348505 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.348510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.348513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.348540 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.351315 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.352229 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.352237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.352242 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.352409 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.352851 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.352858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.352862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.353223 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:19.353377 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:19.353934 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 14:50:19.354179 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:50:19.356921 7124 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.241.61:40459: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb"
member_type: NON_VOTER
last_known_addr {
host: "127.0.241.61"
port: 40459
}
attrs {
promote: true
}
}
dest_uuid: "4002d8e4b9404f0c9d8901c3203d3483"
cas_config_opid_index: -1
I20260619 14:50:19.358085 7124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } }
I20260619 14:50:19.367967 7681 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 04fac3b86b9f46cda4436820eb3e55fb (127.0.241.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } } }
I20260619 14:50:19.369391 7627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:19.372198 7681 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.241.61:40459
I20260619 14:50:19.372661 7681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } } }
I20260619 14:50:19.372946 7681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:19.372989 7585 master.cc:537] Master@127.0.241.61:40459 shutting down...
I20260619 14:50:19.373615 7627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 FOLLOWER]: Refusing update from remote peer 4002d8e4b9404f0c9d8901c3203d3483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 14:50:19.375101 7682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.60.keytab.
Entry for principal oryx/127.0.241.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.60.keytab.
I20260619 14:50:19.379874 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.60:33423
--webserver_interface=127.0.241.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.60.keytab
--principal=oryx/127.0.241.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:33289,127.0.241.61:40459,127.0.241.60:33423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:19.386844 7108 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 14:50:19.387218 7585 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:19.387781 7585 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:19.388059 7585 tablet_replica.cc:330] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: stopping tablet replica
W20260619 14:50:19.388510 7108 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 -> Peer 04fac3b86b9f46cda4436820eb3e55fb (127.0.241.61:40459): Couldn't send request to peer 04fac3b86b9f46cda4436820eb3e55fb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:50:19.404714 7585 master.cc:552] Master@127.0.241.61:40459 shutdown complete.
I20260619 14:50:19.404817 7585 master_runner.cc:305] Clearing existing system tablet
I20260619 14:50:19.405364 7585 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:19.405488 7585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:19.408372 7585 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:50:19.410123 7687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:19.410498 7585 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:19.410902 7585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:50:19.411021 7585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/wal
uuid: "04fac3b86b9f46cda4436820eb3e55fb"
format_stamp: "Formatted at 2026-06-19 14:50:19 on dist-test-slave-p3ml"
server_key: "632b3185363f58531f38cc38d2f3c3dc"
server_key_iv: "c358348c8ba6249e26b97ae29b6df882"
server_key_version: "encryptionkey@0"
I20260619 14:50:19.416139 7585 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:19.422627 7585 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:50:19.422861 7585 log.cc:1196] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 14:50:19.423334 7585 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: Deleting consensus metadata
I20260619 14:50:19.423645 7585 master_runner.cc:315] Copying system tablet from 127.0.241.62:33289
I20260619 14:50:19.428627 7585 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.241.62:33289
I20260619 14:50:19.450943 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.429482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.0.241.62:33289)
0619 14:50:19.429646 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.429652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.436127 (+ 6475us) server_negotiation.cc:407] Connection header received
0619 14:50:19.436327 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.436334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.436431 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.436549 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.437612 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.438759 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.440770 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.441439 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.441564 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.444472 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.444479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.444484 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.444533 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.447982 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.448820 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.448826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.448830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.448969 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.449405 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.449410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.449414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.449752 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:19.449904 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:19.450374 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 14:50:19.450595 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:50:19.450999 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.429442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:56508)
0619 14:50:19.435965 (+ 6523us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.435994 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.436181 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.436608 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.436612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.437138 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:19.437430 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.437444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.438918 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.438924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.440593 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.440614 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.441679 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.441688 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.441935 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.442728 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:19.442759 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:19.444237 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:19.448124 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.448139 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.448143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.448681 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.449086 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.449092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.449096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.449277 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.449880 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:19.449885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:19.450073 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 14:50:19.450234 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:50:19.450546 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6339,"thread_start_us":6197,"threads_started":1}
I20260619 14:50:19.452162 7144 tablet_copy_service.cc:140] P 4002d8e4b9404f0c9d8901c3203d3483: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 04fac3b86b9f46cda4436820eb3e55fb ({username='oryx', principal='oryx/127.0.241.61@KRBTEST.COM'} at 127.0.0.1:56508)
I20260619 14:50:19.452440 7144 tablet_copy_service.cc:161] P 4002d8e4b9404f0c9d8901c3203d3483: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 04fac3b86b9f46cda4436820eb3e55fb at {username='oryx', principal='oryx/127.0.241.61@KRBTEST.COM'} at 127.0.0.1:56508: session id = 04fac3b86b9f46cda4436820eb3e55fb-00000000000000000000000000000000
I20260619 14:50:19.456952 7144 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:50:19.459170 7585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:50:19.466310 7585 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: tablet copy: Starting download of 0 data blocks...
I20260619 14:50:19.466637 7585 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: tablet copy: Starting download of 1 WAL segments...
I20260619 14:50:19.469280 7585 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:50:19.473663 7144 tablet_copy_service.cc:342] P 4002d8e4b9404f0c9d8901c3203d3483: Request end of tablet copy session 04fac3b86b9f46cda4436820eb3e55fb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.241.61@KRBTEST.COM'} at 127.0.0.1:56508
I20260619 14:50:19.473896 7144 tablet_copy_service.cc:433] P 4002d8e4b9404f0c9d8901c3203d3483: ending tablet copy session 04fac3b86b9f46cda4436820eb3e55fb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 04fac3b86b9f46cda4436820eb3e55fb
I20260619 14:50:19.484683 7585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:19.487393 7694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:19.498986 7697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:19.500036 7695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:19.504832 7585 server_base.cc:1034] running on GCE node
I20260619 14:50:19.505203 7585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:19.505833 7585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:19.507119 7585 hybrid_clock.cc:648] HybridClock initialized: now 1781880619507054 us; error 34 us; skew 500 ppm
I20260619 14:50:19.507360 7585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:19 dist-test-slave-p3ml krb5kdc[7079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:19.509624 7585 init.cc:377] Logged in from keytab as oryx/127.0.241.61@KRBTEST.COM (short username oryx)
I20260619 14:50:19.512283 7585 webserver.cc:460] Webserver started at http://127.0.241.61:32847/ using document root <none> and password file <none>
I20260619 14:50:19.512634 7585 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:19.512715 7585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:19.515803 7585 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260619 14:50:19.517982 7704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:19.518467 7585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 14:50:19.518641 7585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/wal
uuid: "04fac3b86b9f46cda4436820eb3e55fb"
format_stamp: "Formatted at 2026-06-19 14:50:19 on dist-test-slave-p3ml"
server_key: "632b3185363f58531f38cc38d2f3c3dc"
server_key_iv: "c358348c8ba6249e26b97ae29b6df882"
server_key_version: "encryptionkey@0"
I20260619 14:50:19.518884 7585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:19.537704 7585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:19.541838 7585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:19.577526 7585 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.61:40459
I20260619 14:50:19.578575 7585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
I20260619 14:50:19.577564 7755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.61:40459 every 8 connection(s)
I20260619 14:50:19.590665 7756 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:50:19.594720 7756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: Bootstrap starting.
I20260619 14:50:19.612531 7756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:50:19.613176 7756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: Bootstrap complete.
I20260619 14:50:19.614351 7756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } }
I20260619 14:50:19.614775 7756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 04fac3b86b9f46cda4436820eb3e55fb, State: Initialized, Role: LEARNER
I20260619 14:50:19.615060 7756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } }
I20260619 14:50:19.616056 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } } }
I20260619 14:50:19.616397 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: This master's current role is: LEARNER
I20260619 14:50:19.616570 7756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:19.629655 7770 catalog_manager.cc:1259] Loaded cluster ID: d1e6b37ed3e14281bc735000d4ab54a6
I20260619 14:50:19.629763 7770 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: loading cluster ID for follower catalog manager: success
I20260619 14:50:19.633306 7770 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: acquiring CA information for follower catalog manager: success
I20260619 14:50:19.635726 7770 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:50:19.760700 7580 heartbeater.cc:503] Master 127.0.241.62:33289 was elected leader, sending a full tablet report...
W20260619 14:50:19.787808 7684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:19.788084 7684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:19.788250 7684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:19.796317 7684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:19.796500 7684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:19.796602 7684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:19.796710 7684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:19.796801 7684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:19.802146 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.796275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:40459 (local address 127.0.0.1:42698)
0619 14:50:19.796477 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.796494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.796596 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.797411 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.797417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.797444 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:19.797638 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.797647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.798899 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.798904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.800205 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.800217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.801424 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.801429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.801632 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.801697 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 14:50:19.801830 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 14:50:19.801952 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:50:19.802363 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.796429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.0.241.61:40459)
0619 14:50:19.797090 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.797096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.797118 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:50:19.797183 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.797187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.797251 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.797380 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:19.797780 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.798751 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.800346 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.801266 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.801405 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.801647 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:19.801876 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 14:50:19.802033 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":367,"threads_started":1}
I20260619 14:50:19.803279 7731 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 14:50:19.803869 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4002d8e4b9404f0c9d8901c3203d3483. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } } }
I20260619 14:50:19.804140 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: This master's current role is: LEARNER
I20260619 14:50:19.807818 7684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:33289,127.0.241.61:40459,127.0.241.60:33423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.60.keytab
--principal=oryx/127.0.241.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.60:33423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.241.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:19.809047 7684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:19.810314 7684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:19.822996 7776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:19.823091 7777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:19.823613 7779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:19.824175 7684 server_base.cc:1034] running on GCE node
I20260619 14:50:19.825027 7684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:19.826475 7684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:19.827692 7684 hybrid_clock.cc:648] HybridClock initialized: now 1781880619827630 us; error 50 us; skew 500 ppm
I20260619 14:50:19.828162 7684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:19 dist-test-slave-p3ml krb5kdc[7079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:19.831564 7684 init.cc:377] Logged in from keytab as oryx/127.0.241.60@KRBTEST.COM (short username oryx)
I20260619 14:50:19.833127 7684 webserver.cc:460] Webserver started at http://127.0.241.60:41987/ using document root <none> and password file <none>
I20260619 14:50:19.833817 7684 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:19.833976 7684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:19.834306 7684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:19.837572 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data/instance:
uuid: "cd5adced693746f4b6b64150310bc02d"
format_stamp: "Formatted at 2026-06-19 14:50:19 on dist-test-slave-p3ml"
server_key: "a67fcb022184a8c3da358d0d2d8149b4"
server_key_iv: "027e5730463590486fcc88bb446ceb1d"
server_key_version: "encryptionkey@0"
I20260619 14:50:19.838431 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/wal/instance:
uuid: "cd5adced693746f4b6b64150310bc02d"
format_stamp: "Formatted at 2026-06-19 14:50:19 on dist-test-slave-p3ml"
server_key: "a67fcb022184a8c3da358d0d2d8149b4"
server_key_iv: "027e5730463590486fcc88bb446ceb1d"
server_key_version: "encryptionkey@0"
I20260619 14:50:19.844094 7684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 14:50:19.848198 7786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:19.849506 7684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:19.849681 7684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/wal
uuid: "cd5adced693746f4b6b64150310bc02d"
format_stamp: "Formatted at 2026-06-19 14:50:19 on dist-test-slave-p3ml"
server_key: "a67fcb022184a8c3da358d0d2d8149b4"
server_key_iv: "027e5730463590486fcc88bb446ceb1d"
server_key_version: "encryptionkey@0"
I20260619 14:50:19.849895 7684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:19.866557 7684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:19.870867 7684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:19.871156 7684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:19.890878 7684 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.60:33423
I20260619 14:50:19.890919 7837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.60:33423 every 8 connection(s)
I20260619 14:50:19.892390 7684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
I20260619 14:50:19.893726 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 7684
I20260619 14:50:19.896899 7838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:19.903339 7838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:50:19 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.241.60@KRBTEST.COM
Jun 19 14:50:19 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.60@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:19.916633 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.895911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.0.241.60:33423)
0619 14:50:19.896610 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.896620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.896680 (+ 60us) server_negotiation.cc:407] Connection header received
0619 14:50:19.898744 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.898765 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.899122 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.899562 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.900927 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.902245 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.903421 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.904089 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.904272 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.909138 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.909145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.909154 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.909185 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.912816 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.913488 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.913494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.913500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.913643 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.914063 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.914067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.914069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.914508 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:19.914826 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:19.915221 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 14:50:19.915636 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":318,"threads_started":1}
I20260619 14:50:19.920104 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.906078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56516 (local address 127.0.241.62:33289)
0619 14:50:19.906310 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.906319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.906983 (+ 664us) server_negotiation.cc:407] Connection header received
0619 14:50:19.907155 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.907159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.907220 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.907296 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.908481 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.909197 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.911066 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.911512 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.911656 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.914790 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.914795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.914799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.914828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.917413 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.918112 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.918120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.918123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.918283 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.918664 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.918670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.918673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.918918 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:19.919063 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:19.919565 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 14:50:19.919754 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:50:19.920153 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.906020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:56516)
0619 14:50:19.906815 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.906866 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.907043 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.907382 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.907389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.908049 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:19.908342 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.908351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.909331 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.909337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.910917 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.910928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.911694 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.911705 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.911866 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.912417 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:19.912443 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:19.914552 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:19.917537 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.917547 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.917553 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.917993 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.918388 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.918393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.918398 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.918525 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.919036 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:19.919042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:19.919218 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 14:50:19.919469 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 14:50:19.919751 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":396,"threads_started":1}
I20260619 14:50:19.921300 7838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:50:19 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.60@KRBTEST.COM for oryx/127.0.241.61@KRBTEST.COM
I20260619 14:50:19.933194 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.921921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:40459 (local address 127.0.0.1:42700)
0619 14:50:19.922098 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.922116 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.922203 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.922479 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.922483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.922850 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:19.922963 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.922970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.924092 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.924097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.924985 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.924995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.925679 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.925684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.925844 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.926683 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:19.926697 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:19.928448 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:19.931182 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.931185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.931187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.931609 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.931950 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.931953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.931956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.932084 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.932606 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:19.932609 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:19.932711 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 14:50:19.932814 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:50:19.932995 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 14:50:19.933352 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.921984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.0.241.61:40459)
0619 14:50:19.922169 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.922174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.922191 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:50:19.922298 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.922304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.922368 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.922465 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.923109 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.923967 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.925121 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.925532 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.925658 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.928610 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.928614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.928617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.928646 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.931067 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.931702 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.931706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.931708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.931832 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.932222 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.932226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.932228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.932503 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:19.932623 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:19.932898 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 14:50:19.933063 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:50:19.934168 7838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:50:19 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.60@KRBTEST.COM for oryx/127.0.241.60@KRBTEST.COM
I20260619 14:50:19.945246 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.934751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:33423 (local address 127.0.0.1:41076)
0619 14:50:19.934941 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.934958 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.935046 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.935335 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.935338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.935682 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:19.935806 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.935813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.936680 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.936683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.937460 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.937470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.938094 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.938097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.938239 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.938715 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:19.938729 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:19.940588 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:19.943347 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.943350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.943353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.943738 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.944040 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.944043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.944045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:19.944189 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:19.944606 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:19.944609 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:19.944722 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 14:50:19.944817 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 14:50:19.944975 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 14:50:19.945394 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.934987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.0.241.60:33423)
0619 14:50:19.935112 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.935116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.935128 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:50:19.935178 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.935181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.935227 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.935299 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.935915 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.936581 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.937582 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.937969 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.938062 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.940742 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.940747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.940750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.940775 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.943230 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.943834 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.943838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.943840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.943950 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:19.944281 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:19.944284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:19.944286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:19.944516 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:19.944614 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:19.944904 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 14:50:19.945084 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":47}
I20260619 14:50:19.950114 7838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: Bootstrap starting.
I20260619 14:50:19.953138 7838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:19.954113 7838 log.cc:824] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:19.956581 7838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: No bootstrap required, opened a new log
I20260619 14:50:19.963591 7838 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } }
I20260619 14:50:19.963856 7838 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:19.963945 7838 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5adced693746f4b6b64150310bc02d, State: Initialized, Role: FOLLOWER
I20260619 14:50:19.964462 7838 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } }
I20260619 14:50:19.965227 7844 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } } }
I20260619 14:50:19.965939 7844 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:50:19.966166 7838 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:50:19.974918 7855 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:50:19.975055 7855 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:50:19.979377 7684 master_runner.cc:428] Detected that this master cd5adced693746f4b6b64150310bc02d is joining an existing cluster
I20260619 14:50:19.979511 7684 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.241.60:33423
I20260619 14:50:20.003103 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.984544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.0.241.60:33423)
0619 14:50:19.984707 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.984714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.985738 (+ 1024us) server_negotiation.cc:407] Connection header received
0619 14:50:19.985960 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.985964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.986032 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.986094 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.987450 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.988522 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.990653 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.991393 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.991522 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.994987 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.994992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.994997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.995031 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.999370 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.000230 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.000235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.000238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.000420 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.000863 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.000869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.000875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.001430 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:20.001595 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:20.002469 (+ 874us) server_negotiation.cc:299] Negotiation successful
0619 14:50:20.002695 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:50:20.003538 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.983541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.0.241.61:40459)
0619 14:50:19.983734 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.983742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.984840 (+ 1098us) server_negotiation.cc:407] Connection header received
0619 14:50:19.985044 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.985052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.985145 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.985267 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.987011 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.988461 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.991829 (+ 3368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.992676 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.992784 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.996303 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:19.996310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:19.996315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:19.996358 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:19.999812 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.000613 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.000618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.000622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.000769 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.001430 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.001435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.001438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.002177 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:20.002324 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:20.002925 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 14:50:20.003161 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:50:20.004657 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.984885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:33423 (local address 127.0.0.1:41092)
0619 14:50:19.985635 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.985665 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.985829 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.986266 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.986270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.986977 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:19.987279 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.987290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.988725 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.988729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.990484 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.990504 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.991842 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.991848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.992014 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.992757 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:19.993259 (+ 502us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:19.994788 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:19.999536 (+ 4748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.999541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.999544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.000104 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.000555 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.000561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.000564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.000745 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.001682 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:20.001686 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:20.002157 (+ 471us) client_negotiation.cc:769] Sending connection context
0619 14:50:20.003754 (+ 1597us) client_negotiation.cc:240] Negotiation successful
0619 14:50:20.004117 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":7,"spinlock_wait_cycles":30080,"thread_start_us":430,"threads_started":1}
I20260619 14:50:20.004901 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.983568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:40459 (local address 127.0.0.1:42706)
0619 14:50:19.984740 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.984761 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.984905 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.985483 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.985490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.986266 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:19.986793 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.986810 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.988680 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.988689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.991604 (+ 2915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.991628 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.993814 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.993823 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.994048 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:19.994743 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:19.994770 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:19.996091 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:19.999978 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:19.999983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:19.999987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.000459 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.000932 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.000936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.000940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.001276 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.002416 (+ 1140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:20.002420 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:20.002624 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 14:50:20.004025 (+ 1401us) client_negotiation.cc:240] Negotiation successful
0619 14:50:20.004389 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":27,"thread_start_us":853,"threads_started":1}
I20260619 14:50:20.009136 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.985763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:56526)
0619 14:50:19.992806 (+ 7043us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:19.992825 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:19.993488 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:19.994210 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:19.994215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:19.994989 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:19.995387 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.995404 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.997097 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.997105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:19.998983 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:19.998999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.000315 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.000322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.000499 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.001124 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:20.001145 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:20.002688 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:20.006675 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.006680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.006683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.007109 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.007555 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.007559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.007561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.007697 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.008356 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:20.008359 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:20.008513 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 14:50:20.008620 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:50:20.008835 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6868,"thread_start_us":7159,"threads_started":1}
I20260619 14:50:20.009358 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:19.983736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.0.241.62:33289)
0619 14:50:19.984759 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:19.984767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:19.992906 (+ 8139us) server_negotiation.cc:407] Connection header received
0619 14:50:19.993785 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:19.993792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:19.993913 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:19.994032 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:19.995607 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.996878 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:19.999167 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:19.999919 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.000021 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.002915 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:20.002923 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:20.002928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:20.002976 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:20.006507 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.007246 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.007253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.007257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.007422 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.007824 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.007830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.007833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.008216 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:20.008361 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:20.008761 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 14:50:20.009011 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879}
I20260619 14:50:20.022564 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 7185
Jun 19 14:50:20 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.241.1@KRBTEST.COM
Jun 19 14:50:20 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.62@KRBTEST.COM for oryx/127.0.241.60@KRBTEST.COM
I20260619 14:50:20.041893 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.023632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.0.241.1:38131)
0619 14:50:20.024317 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:20.024324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:20.024348 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:50:20.024402 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:20.024407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:20.024613 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:20.024787 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:20.026045 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.027070 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.028432 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.029317 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.029453 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.034465 (+ 5012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:20.034478 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:20.034498 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:20.034542 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:20.038541 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.039368 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.039380 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.039385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.039564 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.039975 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.039980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.039984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.040319 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:20.040660 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:20.040996 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 14:50:20.041270 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":362,"threads_started":1}
I20260619 14:50:20.043327 7308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:20.045656 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.014836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:33423 (local address 127.0.0.1:41102)
0619 14:50:20.015040 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:20.015061 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:20.015178 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:20.015644 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:20.015650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:20.016094 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:20.016353 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:20.016366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.032992 (+ 16626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.033000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.033967 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:20.033985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.034952 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.034959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.035148 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.036022 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:20.036051 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:20.038362 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:20.042534 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.042539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.042543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.043061 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.043545 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.043554 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.043560 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.043761 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.044649 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:20.044656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:20.044836 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 14:50:20.045004 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 14:50:20.045285 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:50:20.046008 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.014909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.0.241.60:33423)
0619 14:50:20.015071 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:20.015075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:20.015127 (+ 52us) server_negotiation.cc:407] Connection header received
0619 14:50:20.015320 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:20.015323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:20.015488 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:20.015617 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:20.031730 (+ 16113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.032834 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.034122 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.034746 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.034890 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.038573 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:20.038582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:20.038590 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:20.038633 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:20.042383 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.043195 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.043200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.043204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.043356 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.043960 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.043968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.043979 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.044437 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:20.044658 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:20.045152 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 14:50:20.045476 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:50:20.048013 7123 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.241.60:33423: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cd5adced693746f4b6b64150310bc02d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.241.60"
port: 33423
}
attrs {
promote: true
}
}
dest_uuid: "4002d8e4b9404f0c9d8901c3203d3483"
cas_config_opid_index: 5
I20260619 14:50:20.049475 7123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } }
I20260619 14:50:20.051342 7682 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cd5adced693746f4b6b64150310bc02d (127.0.241.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
I20260619 14:50:20.052224 7731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 LEARNER]: Refusing update from remote peer 4002d8e4b9404f0c9d8901c3203d3483: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:50:20.056288 7813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:20.060266 7813 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 FOLLOWER]: Refusing update from remote peer 4002d8e4b9404f0c9d8901c3203d3483: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:50:20.061420 7867 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 14:50:20.061929 7682 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.241.60:33423
I20260619 14:50:20.062674 7684 master.cc:537] Master@127.0.241.60:33423 shutting down...
I20260619 14:50:20.077759 7868 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:20.078553 7684 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:20.078154 7731 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cd5adced693746f4b6b64150310bc02d (127.0.241.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
I20260619 14:50:20.079030 7684 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:20.079274 7684 tablet_replica.cc:330] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: stopping tablet replica
I20260619 14:50:20.082703 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
I20260619 14:50:20.083007 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: This master's current role is: LEARNER
I20260619 14:50:20.085336 7682 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483: attempting to promote NON_VOTER 04fac3b86b9f46cda4436820eb3e55fb to VOTER
I20260619 14:50:20.086163 7682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } }
I20260619 14:50:20.086514 7684 master.cc:552] Master@127.0.241.60:33423 shutdown complete.
I20260619 14:50:20.086584 7684 master_runner.cc:305] Clearing existing system tablet
I20260619 14:50:20.086992 7684 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:20.087103 7684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:20.087865 7868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
I20260619 14:50:20.088236 7868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:20.091713 7867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: NON_VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: true } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
I20260619 14:50:20.093861 7867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:20.097121 7731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 LEARNER]: Refusing update from remote peer 4002d8e4b9404f0c9d8901c3203d3483: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 14:50:20.097977 7682 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 14:50:20.100368 7684 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.001s sys 0.009s
I20260619 14:50:20.101080 7871 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 04fac3b86b9f46cda4436820eb3e55fb (127.0.241.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
W20260619 14:50:20.101779 7109 connection.cc:698] client connection to 127.0.241.60:33423 send error: Network error: sendmsg error: Broken pipe (error 32)
I20260619 14:50:20.103755 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.102707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:33423 (local address 127.0.0.1:41118)
0619 14:50:20.103259 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:20.103381 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:33423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439}
W20260619 14:50:20.104290 7109 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 -> Peer cd5adced693746f4b6b64150310bc02d (127.0.241.60:33423): Couldn't send request to peer cd5adced693746f4b6b64150310bc02d. Status: Network error: Client connection negotiation failed: client connection to 127.0.241.60:33423: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:50:20.104074 7730 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 04fac3b86b9f46cda4436820eb3e55fb (127.0.241.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
I20260619 14:50:20.107349 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
I20260619 14:50:20.107601 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:50:20.109628 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
I20260619 14:50:20.110095 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:20.110420 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
I20260619 14:50:20.110679 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:20.111145 7874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:20.111579 7684 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:20.111863 7684 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.001s
I20260619 14:50:20.111969 7684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/wal
uuid: "cd5adced693746f4b6b64150310bc02d"
format_stamp: "Formatted at 2026-06-19 14:50:19 on dist-test-slave-p3ml"
server_key: "a67fcb022184a8c3da358d0d2d8149b4"
server_key_iv: "027e5730463590486fcc88bb446ceb1d"
server_key_version: "encryptionkey@0"
I20260619 14:50:20.115273 7684 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:20.120388 7684 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:50:20.120555 7684 log.cc:1196] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 14:50:20.120973 7684 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: Deleting consensus metadata
I20260619 14:50:20.121213 7684 master_runner.cc:315] Copying system tablet from 127.0.241.62:33289
I20260619 14:50:20.123993 7684 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.241.62:33289
I20260619 14:50:20.136655 7315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.124783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.0.241.62:33289)
0619 14:50:20.125054 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:20.125062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:20.125551 (+ 489us) server_negotiation.cc:407] Connection header received
0619 14:50:20.125760 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:20.125765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:20.125840 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:20.125955 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:20.126882 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.127813 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.129035 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.129488 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.129617 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.131371 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:20.131377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:20.131383 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:20.131482 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:20.134059 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.134667 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.134673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.134677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.134834 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.135205 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.135210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.135214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.135542 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:20.135693 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:20.136075 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 14:50:20.136283 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 14:50:20.136765 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.124724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:56536)
0619 14:50:20.125442 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:20.125461 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:20.125620 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:20.125981 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:20.125986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:20.126485 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:20.126732 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:20.126744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.127943 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.127948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.128877 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:20.128887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.129629 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.129634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.129821 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.130454 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:20.130471 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:20.131168 (+ 697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:20.134193 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.134201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.134205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.134541 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.134948 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.134954 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.134958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.135095 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.135655 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:20.135660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:20.135836 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 14:50:20.135976 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 14:50:20.136311 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":460,"threads_started":1}
I20260619 14:50:20.137627 7144 tablet_copy_service.cc:140] P 4002d8e4b9404f0c9d8901c3203d3483: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cd5adced693746f4b6b64150310bc02d ({username='oryx', principal='oryx/127.0.241.60@KRBTEST.COM'} at 127.0.0.1:56536)
I20260619 14:50:20.137789 7144 tablet_copy_service.cc:161] P 4002d8e4b9404f0c9d8901c3203d3483: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cd5adced693746f4b6b64150310bc02d at {username='oryx', principal='oryx/127.0.241.60@KRBTEST.COM'} at 127.0.0.1:56536: session id = cd5adced693746f4b6b64150310bc02d-00000000000000000000000000000000
I20260619 14:50:20.140374 7144 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:50:20.142019 7684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:50:20.146490 7684 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: tablet copy: Starting download of 0 data blocks...
I20260619 14:50:20.146761 7684 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: tablet copy: Starting download of 1 WAL segments...
I20260619 14:50:20.148847 7684 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:50:20.158555 7144 tablet_copy_service.cc:342] P 4002d8e4b9404f0c9d8901c3203d3483: Request end of tablet copy session cd5adced693746f4b6b64150310bc02d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.241.60@KRBTEST.COM'} at 127.0.0.1:56536
I20260619 14:50:20.158692 7144 tablet_copy_service.cc:433] P 4002d8e4b9404f0c9d8901c3203d3483: ending tablet copy session cd5adced693746f4b6b64150310bc02d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cd5adced693746f4b6b64150310bc02d
I20260619 14:50:20.171157 7684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:20.173800 7881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:20.180585 7882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:20.188846 7684 server_base.cc:1034] running on GCE node
W20260619 14:50:20.189136 7884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:20.189558 7684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:20.190152 7684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:20.191462 7684 hybrid_clock.cc:648] HybridClock initialized: now 1781880620191354 us; error 64 us; skew 500 ppm
I20260619 14:50:20.191689 7684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:20 dist-test-slave-p3ml krb5kdc[7079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880620, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:20.193519 7684 init.cc:377] Logged in from keytab as oryx/127.0.241.60@KRBTEST.COM (short username oryx)
I20260619 14:50:20.196318 7684 webserver.cc:460] Webserver started at http://127.0.241.60:40175/ using document root <none> and password file <none>
I20260619 14:50:20.196658 7684 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:20.196774 7684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:20.199585 7684 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:50:20.201244 7891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:20.201654 7684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 14:50:20.201855 7684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/wal
uuid: "cd5adced693746f4b6b64150310bc02d"
format_stamp: "Formatted at 2026-06-19 14:50:19 on dist-test-slave-p3ml"
server_key: "a67fcb022184a8c3da358d0d2d8149b4"
server_key_iv: "027e5730463590486fcc88bb446ceb1d"
server_key_version: "encryptionkey@0"
I20260619 14:50:20.202039 7684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:20.221531 7684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:20.226807 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 7185
I20260619 14:50:20.227185 7684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:20.250982 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:38131
--local_ip_for_outbound_sockets=127.0.241.1
--tserver_master_addrs=127.0.241.62:33289
--webserver_port=45319
--webserver_interface=127.0.241.1
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab
--principal=oryx/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.241.62:33289,127.0.241.61:40459
--tserver_master_addrs=127.0.241.62:33289,127.0.241.61:40459,127.0.241.60:33423 with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:50:20.270586 7684 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.60:33423
I20260619 14:50:20.270622 7943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.60:33423 every 8 connection(s)
I20260619 14:50:20.271544 7684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
I20260619 14:50:20.278404 7944 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:50:20.284276 7944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: Bootstrap starting.
I20260619 14:50:20.300318 7944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:50:20.300745 7944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: Bootstrap complete.
I20260619 14:50:20.301558 7944 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } }
I20260619 14:50:20.301890 7944 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd5adced693746f4b6b64150310bc02d, State: Initialized, Role: LEARNER
I20260619 14:50:20.302163 7944 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } }
I20260619 14:50:20.305526 7947 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
I20260619 14:50:20.305833 7944 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:20.305995 7947 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: This master's current role is: LEARNER
I20260619 14:50:20.331161 7958 catalog_manager.cc:1259] Loaded cluster ID: d1e6b37ed3e14281bc735000d4ab54a6
I20260619 14:50:20.331290 7958 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: loading cluster ID for follower catalog manager: success
I20260619 14:50:20.333600 7958 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: acquiring CA information for follower catalog manager: success
I20260619 14:50:20.335238 7958 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:50:20.490427 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.483425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:33423 (local address 127.0.0.1:41124)
0619 14:50:20.483694 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:20.483711 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:20.483815 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:20.484542 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:20.484547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:20.484595 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:20.484780 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:20.484791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.486152 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.486161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.488195 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:20.488210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.489445 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.489452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.489689 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.489739 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 14:50:20.489886 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 14:50:20.490094 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260619 14:50:20.490450 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.483632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.0.241.60:33423)
0619 14:50:20.484213 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:20.484217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:20.484232 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:50:20.484287 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:20.484300 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:20.484361 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:20.484456 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:20.484944 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.485978 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.488360 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.489288 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.489384 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.489549 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:20.489944 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 14:50:20.490079 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":334,"threads_started":1}
I20260619 14:50:20.491497 7918 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 14:50:20.492070 7947 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4002d8e4b9404f0c9d8901c3203d3483. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
I20260619 14:50:20.492367 7947 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: This master's current role is: LEARNER
W20260619 14:50:20.605834 7920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:20.606082 7920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:20.606246 7920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:20.613862 7920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:20.613963 7920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:20.614138 7920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:50:20.623963 7920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab
--principal=oryx/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:38131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=45319
--tserver_master_addrs=127.0.241.62:33289,127.0.241.61:40459,127.0.241.60:33423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:20.625306 7920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:20.626557 7920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:20.639751 7964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:20.639775 7967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:20.640437 7965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:20.646219 7920 server_base.cc:1034] running on GCE node
I20260619 14:50:20.646867 7920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:20.647782 7920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:20.648993 7920 hybrid_clock.cc:648] HybridClock initialized: now 1781880620648933 us; error 53 us; skew 500 ppm
I20260619 14:50:20.649338 7920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:20 dist-test-slave-p3ml krb5kdc[7079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880620, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:20.652422 7920 init.cc:377] Logged in from keytab as oryx/127.0.241.1@KRBTEST.COM (short username oryx)
I20260619 14:50:20.653769 7920 webserver.cc:460] Webserver started at http://127.0.241.1:45319/ using document root <none> and password file <none>
I20260619 14:50:20.654420 7920 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:20.654526 7920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:20.660449 7920 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:50:20.662992 7974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:20.664343 7920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:50:20.664512 7920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "6b840e7d2eb14f1e9cc4094eecbe287b"
format_stamp: "Formatted at 2026-06-19 14:50:17 on dist-test-slave-p3ml"
server_key: "0bbf4cf58301f2616a6ce70628f0fc02"
server_key_iv: "932536dd458e1c7e682318eebf2788d0"
server_key_version: "encryptionkey@0"
I20260619 14:50:20.665446 7920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:20.726966 7920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:20.731506 7920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:20.731772 7920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:20.732728 7920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:20.734061 7920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:20.734179 7920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:20.734311 7920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:20.734407 7920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:20.768600 7920 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:38131
I20260619 14:50:20.768682 8086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:38131 every 8 connection(s)
I20260619 14:50:20.770172 7920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:20.777741 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 7920
I20260619 14:50:20.778110 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 7319
Jun 19 14:50:20 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.241.2@KRBTEST.COM
Jun 19 14:50:20 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880620, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.60@KRBTEST.COM
Jun 19 14:50:20 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880620, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.61@KRBTEST.COM
Jun 19 14:50:20 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880620, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:20.825138 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.773825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:48759 (local address 127.0.241.60:33423)
0619 14:50:20.774085 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:20.774093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:20.800691 (+ 26598us) server_negotiation.cc:407] Connection header received
0619 14:50:20.800961 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:20.800966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:20.801044 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:20.801112 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:20.804016 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.805091 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.807999 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.808891 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.809009 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.816282 (+ 7273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:20.816289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:20.816294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:20.816363 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:20.821073 (+ 4710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.822086 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.822093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.822096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.822258 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.822814 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.822820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.822824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.823222 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:20.823361 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:20.824405 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0619 14:50:20.824669 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260619 14:50:20.826525 8094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.774899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:33423 (local address 127.0.241.1:48759)
0619 14:50:20.800588 (+ 25689us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:20.800607 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:20.800787 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:20.801578 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:20.801589 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:20.802511 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:20.803575 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:20.803605 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.805287 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.805292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.807808 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:20.807826 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.809333 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.809339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.809614 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.812294 (+ 2680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:20.812328 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:20.816019 (+ 3691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:20.821212 (+ 5193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.821223 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.821242 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.821925 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.822429 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.822434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.822438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.822665 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.823479 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:20.823488 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:20.823925 (+ 437us) client_negotiation.cc:769] Sending connection context
0619 14:50:20.825049 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0619 14:50:20.825522 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25567,"thread_start_us":28808,"threads_started":1}
I20260619 14:50:20.828425 8087 heartbeater.cc:347] Connected to a master server at 127.0.241.60:33423
I20260619 14:50:20.828783 8087 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:20.829712 8097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.779281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.0.241.2:38485)
0619 14:50:20.797379 (+ 18098us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:20.797389 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:20.798351 (+ 962us) server_negotiation.cc:407] Connection header received
0619 14:50:20.798427 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:20.798436 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:20.798710 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:20.798870 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:20.801141 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.802753 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.805899 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.806676 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.806792 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.816158 (+ 9366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:20.816173 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:20.816190 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:20.816238 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:20.823081 (+ 6843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.824034 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.824043 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.824047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.824214 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.824755 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.824763 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.824769 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.825203 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:20.825495 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:20.827851 (+ 2356us) server_negotiation.cc:299] Negotiation successful
0619 14:50:20.828104 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17923,"thread_start_us":18100,"threads_started":1}
I20260619 14:50:20.829818 8087 heartbeater.cc:511] Master 127.0.241.60:33423 requested a full tablet report, sending...
I20260619 14:50:20.830997 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.773512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.1:57831)
0619 14:50:20.811673 (+ 38161us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:20.811697 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:20.811848 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:20.812576 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:20.812581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:20.813255 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:20.813709 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:20.813730 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.814926 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.814935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.817008 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:20.817024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.818238 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.818247 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.818506 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.819135 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:20.819153 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:20.822439 (+ 3286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:20.827554 (+ 5115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.827558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.827561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.828158 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.828600 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.828605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.828609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.828820 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.829410 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:20.829415 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:20.829739 (+ 324us) client_negotiation.cc:769] Sending connection context
0619 14:50:20.830187 (+ 448us) client_negotiation.cc:240] Negotiation successful
0619 14:50:20.830478 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37980,"mutex_wait_us":1794,"spinlock_wait_cycles":2688,"thread_start_us":24243,"threads_started":1}
I20260619 14:50:20.831117 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.773512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:40459 (local address 127.0.241.1:39799)
0619 14:50:20.796450 (+ 22938us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:20.796521 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:20.799009 (+ 2488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:20.810913 (+ 11904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:20.810922 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:20.811492 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:20.811838 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:20.811848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.813423 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.813429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.815509 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:20.815529 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.816693 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.816699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:20.818521 (+ 1822us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.819601 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:20.819629 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:20.822351 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:20.827683 (+ 5332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.827687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.827691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.828203 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.828600 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:20.828605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:20.828609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:20.828823 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:20.829476 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:20.829483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:20.829741 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 14:50:20.830401 (+ 660us) client_negotiation.cc:240] Negotiation successful
0619 14:50:20.830678 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22535,"mutex_wait_us":1633,"spinlock_wait_cycles":1267968,"thread_start_us":37867,"threads_started":1}
I20260619 14:50:20.831621 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.772928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:57831 (local address 127.0.241.62:33289)
0619 14:50:20.811300 (+ 38372us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:20.811306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:20.812028 (+ 722us) server_negotiation.cc:407] Connection header received
0619 14:50:20.812103 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:20.812112 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:20.812213 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:20.812337 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:20.813940 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.814763 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.817163 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.818002 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.818158 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.822628 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:20.822635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:20.822639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:20.822676 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:20.827347 (+ 4671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.828296 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.828303 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.828306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.828473 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.828956 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.828962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.828965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.829276 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:20.829421 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:20.830854 (+ 1433us) server_negotiation.cc:299] Negotiation successful
0619 14:50:20.831104 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38241,"thread_start_us":38133,"threads_started":1}
I20260619 14:50:20.832324 7442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:20.832571 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:20.772879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:39799 (local address 127.0.241.61:40459)
0619 14:50:20.810507 (+ 37628us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:20.810515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:20.810542 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:50:20.810600 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:20.810605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:20.810694 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:20.810815 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:20.811999 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.813242 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.815687 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:20.816321 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:20.816446 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:20.822581 (+ 6135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:20.822587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:20.822591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:20.822628 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:20.827558 (+ 4930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.828319 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.828324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.828327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.828473 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:20.828956 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:20.828962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:20.828965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:20.829276 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:20.829416 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:20.831302 (+ 1886us) server_negotiation.cc:299] Negotiation successful
0619 14:50:20.831914 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37490,"thread_start_us":37387,"threads_started":1}
I20260619 14:50:20.833122 7908 ts_manager.cc:194] Registered new tserver with Master: 6b840e7d2eb14f1e9cc4094eecbe287b (127.0.241.1:38131)
I20260619 14:50:20.833539 8089 heartbeater.cc:347] Connected to a master server at 127.0.241.61:40459
I20260619 14:50:20.833647 8088 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33289
I20260619 14:50:20.833681 8089 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:20.833796 8088 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:20.834233 8089 heartbeater.cc:511] Master 127.0.241.61:40459 requested a full tablet report, sending...
I20260619 14:50:20.834316 8088 heartbeater.cc:511] Master 127.0.241.62:33289 requested a full tablet report, sending...
I20260619 14:50:20.835376 7123 ts_manager.cc:194] Re-registered known tserver with Master: 6b840e7d2eb14f1e9cc4094eecbe287b (127.0.241.1:38131)
I20260619 14:50:20.836278 7123 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:57831
I20260619 14:50:20.836275 7721 ts_manager.cc:194] Registered new tserver with Master: 6b840e7d2eb14f1e9cc4094eecbe287b (127.0.241.1:38131)
I20260619 14:50:20.922012 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 7319
I20260619 14:50:20.938748 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:38485
--local_ip_for_outbound_sockets=127.0.241.2
--tserver_master_addrs=127.0.241.62:33289
--webserver_port=43339
--webserver_interface=127.0.241.2
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab
--principal=oryx/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.241.62:33289,127.0.241.61:40459
--tserver_master_addrs=127.0.241.62:33289,127.0.241.61:40459,127.0.241.60:33423 with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:50:21.027539 7871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: NON_VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: true } } }
I20260619 14:50:21.027885 7871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:21.029026 7871 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483: attempting to promote NON_VOTER cd5adced693746f4b6b64150310bc02d to VOTER
I20260619 14:50:21.029718 7871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } }
I20260619 14:50:21.031466 7918 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 LEARNER]: Refusing update from remote peer 4002d8e4b9404f0c9d8901c3203d3483: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 14:50:21.031692 7730 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 FOLLOWER]: Refusing update from remote peer 4002d8e4b9404f0c9d8901c3203d3483: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 14:50:21.032174 7871 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 14:50:21.032627 8101 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 14:50:21.035602 7871 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cd5adced693746f4b6b64150310bc02d (127.0.241.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } } }
I20260619 14:50:21.036407 7730 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cd5adced693746f4b6b64150310bc02d (127.0.241.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } } }
I20260619 14:50:21.038795 7871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } } }
I20260619 14:50:21.039052 7871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:21.039341 8100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } } }
I20260619 14:50:21.039637 8100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:21.039937 8100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } } }
I20260619 14:50:21.040100 8100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:21.040233 7918 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cd5adced693746f4b6b64150310bc02d (127.0.241.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } } }
I20260619 14:50:21.044194 8104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } } }
I20260619 14:50:21.044678 8104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:50:21.046231 8105 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4002d8e4b9404f0c9d8901c3203d3483" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } } }
I20260619 14:50:21.046617 8105 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: This master's current role is: FOLLOWER
W20260619 14:50:21.264371 8099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:21.264688 8099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:21.264854 8099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:21.272089 8099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:21.272223 8099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:21.272367 8099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:50:21.281564 8099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab
--principal=oryx/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:38485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=43339
--tserver_master_addrs=127.0.241.62:33289,127.0.241.61:40459,127.0.241.60:33423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:21.282720 8099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:21.283962 8099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:21.298110 8110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:21.298017 8111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:21.298044 8113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:21.298530 8099 server_base.cc:1034] running on GCE node
I20260619 14:50:21.299160 8099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:21.300118 8099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:21.301345 8099 hybrid_clock.cc:648] HybridClock initialized: now 1781880621301298 us; error 42 us; skew 500 ppm
I20260619 14:50:21.301671 8099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:21 dist-test-slave-p3ml krb5kdc[7079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880621, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:21.305042 8099 init.cc:377] Logged in from keytab as oryx/127.0.241.2@KRBTEST.COM (short username oryx)
I20260619 14:50:21.306552 8099 webserver.cc:460] Webserver started at http://127.0.241.2:43339/ using document root <none> and password file <none>
I20260619 14:50:21.307360 8099 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:21.307536 8099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:21.312614 8099 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:21.315251 8120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:21.316746 8099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:21.316913 8099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "a6e46bb4f46e4d309506a4167f43343f"
format_stamp: "Formatted at 2026-06-19 14:50:18 on dist-test-slave-p3ml"
server_key: "72d117364a365176047c5b837507efe8"
server_key_iv: "a7252824d4f6d6b667b1ecb31b1abe18"
server_key_version: "encryptionkey@0"
I20260619 14:50:21.317894 8099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:21.337013 8099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:21.341437 8099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:21.341809 8099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:21.342769 8099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:21.344177 8099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:21.344270 8099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:21.344422 8099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:21.344537 8099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:21.377842 8099 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:38485
I20260619 14:50:21.377951 8232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:38485 every 8 connection(s)
I20260619 14:50:21.379248 8099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:21.388571 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 8099
I20260619 14:50:21.388854 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 7452
Jun 19 14:50:21 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880621, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.60@KRBTEST.COM
Jun 19 14:50:21 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.241.3@KRBTEST.COM
I20260619 14:50:21.420113 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.382011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:58981 (local address 127.0.241.60:33423)
0619 14:50:21.382528 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:21.382534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:21.400321 (+ 17787us) server_negotiation.cc:407] Connection header received
0619 14:50:21.400505 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:21.400511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:21.400585 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:21.400706 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:21.402464 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.403867 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.406533 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.407198 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.407317 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:21.411935 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:21.411944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:21.411953 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:21.412006 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:21.415871 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:21.416838 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:21.416845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:21.416852 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:21.416976 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:21.417397 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:21.417402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:21.417405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:21.417910 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:21.418044 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:21.419336 (+ 1292us) server_negotiation.cc:299] Negotiation successful
0619 14:50:21.419602 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":274,"threads_started":1}
Jun 19 14:50:21 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880621, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.61@KRBTEST.COM
I20260619 14:50:21.422703 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.382675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:33423 (local address 127.0.241.2:58981)
0619 14:50:21.400210 (+ 17535us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:21.400225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:21.400370 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:21.400956 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:21.400964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:21.401637 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:21.402282 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:21.402308 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.404052 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.404057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:21.406348 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:21.406363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.407625 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.407633 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:21.407894 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:21.408669 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:21.408699 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:21.411696 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:21.416022 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:21.416030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:21.416046 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:21.416702 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:21.417085 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:21.417090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:21.417094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:21.417279 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:21.418119 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:21.418128 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:21.418591 (+ 463us) client_negotiation.cc:769] Sending connection context
0619 14:50:21.418914 (+ 323us) client_negotiation.cc:240] Negotiation successful
0619 14:50:21.420794 (+ 1880us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17381,"mutex_wait_us":63,"thread_start_us":30693,"threads_started":1}
I20260619 14:50:21.424964 8233 heartbeater.cc:347] Connected to a master server at 127.0.241.60:33423
I20260619 14:50:21.425324 8233 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:21.426169 8233 heartbeater.cc:511] Master 127.0.241.60:33423 requested a full tablet report, sending...
I20260619 14:50:21.427810 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.390421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.0.241.3:44993)
0619 14:50:21.402898 (+ 12477us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:21.402908 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:21.403594 (+ 686us) server_negotiation.cc:407] Connection header received
0619 14:50:21.403678 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:21.403686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:21.403909 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:21.404051 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:21.406631 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.408086 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.410004 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.410977 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.411091 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:21.419572 (+ 8481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:21.419586 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:21.419601 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:21.419638 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:21.424301 (+ 4663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:21.425141 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:21.425149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:21.425153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:21.425318 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:21.425824 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:21.425831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:21.425836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:21.426244 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:21.426537 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:21.427098 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 14:50:21.427365 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12320,"thread_start_us":12180,"threads_started":1}
I20260619 14:50:21.427917 7908 ts_manager.cc:194] Registered new tserver with Master: a6e46bb4f46e4d309506a4167f43343f (127.0.241.2:38485)
I20260619 14:50:21.429136 7575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:21.429598 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.382975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:52105 (local address 127.0.241.61:40459)
0619 14:50:21.411815 (+ 28840us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:21.411822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:21.411840 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:50:21.411897 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:21.411902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:21.411972 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:21.412088 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:21.413876 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.415368 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.417495 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.418154 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.420543 (+ 2389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:21.422622 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:21.422628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:21.422633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:21.422683 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:21.426538 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:21.427231 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:21.427236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:21.427240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:21.427373 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:21.427845 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:21.427851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:21.427858 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:21.428216 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:21.428355 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:21.428906 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 14:50:21.429132 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28710,"thread_start_us":28598,"threads_started":1}
I20260619 14:50:21.430889 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.382675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:40459 (local address 127.0.241.2:52105)
0619 14:50:21.394252 (+ 11577us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:21.394314 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:21.398877 (+ 4563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:21.412107 (+ 13230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:21.412112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:21.412556 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:21.412827 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:21.412841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.415577 (+ 2736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.415583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:21.417276 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:21.417295 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.418384 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.418393 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:21.418644 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:21.419265 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:21.419287 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:21.422282 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:21.426664 (+ 4382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:21.426678 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:21.426682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:21.427091 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:21.427524 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:21.427529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:21.427533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:21.427693 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:21.428353 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:21.428359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:21.428495 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 14:50:21.428643 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 14:50:21.430304 (+ 1661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11177,"thread_start_us":17135,"threads_started":1}
Jun 19 14:50:21 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880621, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:21.432094 8235 heartbeater.cc:347] Connected to a master server at 127.0.241.61:40459
I20260619 14:50:21.432248 8235 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:21.432747 8235 heartbeater.cc:511] Master 127.0.241.61:40459 requested a full tablet report, sending...
I20260619 14:50:21.433933 7721 ts_manager.cc:194] Registered new tserver with Master: a6e46bb4f46e4d309506a4167f43343f (127.0.241.2:38485)
I20260619 14:50:21.438805 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.383328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:51313 (local address 127.0.241.62:33289)
0619 14:50:21.412389 (+ 29061us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:21.412397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:21.413973 (+ 1576us) server_negotiation.cc:407] Connection header received
0619 14:50:21.414101 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:21.414108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:21.414186 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:21.414296 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:21.418768 (+ 4472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.420082 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.422517 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.423163 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.423257 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:21.432803 (+ 9546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:21.432830 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:21.432836 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:21.432887 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:21.436302 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:21.437077 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:21.437082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:21.437084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:21.437203 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:21.437615 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:21.437618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:21.437620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:21.437862 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:21.437960 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:21.438396 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 14:50:21.438543 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28923,"thread_start_us":28017,"threads_started":1}
I20260619 14:50:21.439057 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.383387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.2:51313)
0619 14:50:21.413861 (+ 30474us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:21.413879 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:21.413996 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:21.417550 (+ 3554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:21.417556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:21.418039 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:21.418530 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:21.418542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.420679 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.420684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:21.422315 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:21.422327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:21.423594 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:21.423603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:21.423835 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:21.424375 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:21.424400 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:21.432591 (+ 8191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:21.436440 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:21.436446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:21.436451 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:21.436946 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:21.437326 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:21.437330 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:21.437333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:21.437509 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:21.437986 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:21.437992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:21.438162 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 14:50:21.438306 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 14:50:21.438630 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30349,"thread_start_us":12930,"threads_started":1}
I20260619 14:50:21.439801 8234 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33289
I20260619 14:50:21.439950 8234 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:21.440335 8234 heartbeater.cc:511] Master 127.0.241.62:33289 requested a full tablet report, sending...
I20260619 14:50:21.441215 7123 ts_manager.cc:194] Re-registered known tserver with Master: a6e46bb4f46e4d309506a4167f43343f (127.0.241.2:38485)
I20260619 14:50:21.442000 7123 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:51313
I20260619 14:50:21.513093 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 7452
I20260619 14:50:21.529567 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:44993
--local_ip_for_outbound_sockets=127.0.241.3
--tserver_master_addrs=127.0.241.62:33289
--webserver_port=41279
--webserver_interface=127.0.241.3
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab
--principal=oryx/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.241.62:33289,127.0.241.61:40459
--tserver_master_addrs=127.0.241.62:33289,127.0.241.61:40459,127.0.241.60:33423 with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:50:21.838477 8088 heartbeater.cc:503] Master 127.0.241.62:33289 was elected leader, sending a full tablet report...
W20260619 14:50:21.849387 8246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:21.849669 8246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:21.849836 8246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:21.857389 8246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:21.857527 8246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:21.857697 8246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:50:21.866822 8246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab
--principal=oryx/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:44993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=41279
--tserver_master_addrs=127.0.241.62:33289,127.0.241.61:40459,127.0.241.60:33423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:21.868119 8246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:21.869611 8246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:21.882642 8254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:21.882794 8252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:21.882807 8251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:21.883594 8246 server_base.cc:1034] running on GCE node
I20260619 14:50:21.884207 8246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:21.885123 8246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:21.886335 8246 hybrid_clock.cc:648] HybridClock initialized: now 1781880621886294 us; error 31 us; skew 500 ppm
I20260619 14:50:21.886687 8246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:21 dist-test-slave-p3ml krb5kdc[7079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880621, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:21.889730 8246 init.cc:377] Logged in from keytab as oryx/127.0.241.3@KRBTEST.COM (short username oryx)
I20260619 14:50:21.891151 8246 webserver.cc:460] Webserver started at http://127.0.241.3:41279/ using document root <none> and password file <none>
I20260619 14:50:21.891875 8246 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:21.892022 8246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:21.897128 8246 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:50:21.899897 8261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:21.901113 8246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:50:21.901288 8246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "8cac0c45467f435a83c18d10cc73685a"
format_stamp: "Formatted at 2026-06-19 14:50:18 on dist-test-slave-p3ml"
server_key: "82a2c7eba532c94b22d00d8c031600ff"
server_key_iv: "e213d2489b1d7f02d12bde6c24ebb933"
server_key_version: "encryptionkey@0"
I20260619 14:50:21.902201 8246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:21.920440 8246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:21.924680 8246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:21.924901 8246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:21.925748 8246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:21.927000 8246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:21.927083 8246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:21.927170 8246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:21.927224 8246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:21.958711 8246 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:44993
I20260619 14:50:21.958750 8373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:44993 every 8 connection(s)
I20260619 14:50:21.960350 8246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:21.968310 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 8246
Jun 19 14:50:22 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880621, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
Jun 19 14:50:22 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880621, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.61@KRBTEST.COM
Jun 19 14:50:22 dist-test-slave-p3ml krb5kdc[7079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880621, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.60@KRBTEST.COM
I20260619 14:50:22.037375 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.963266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:44019 (local address 127.0.241.61:40459)
0619 14:50:22.000010 (+ 36744us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.000018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.000033 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:50:22.000082 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.000086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.000178 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.000284 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:22.007452 (+ 7168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.008663 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.015089 (+ 6426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.016024 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.016189 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.023410 (+ 7221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:22.023417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:22.023422 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:22.023498 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:22.032198 (+ 8700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.033058 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.033065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.033072 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.033248 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.034377 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.034382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.034385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.034746 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:22.034892 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.036728 (+ 1836us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.036951 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36593,"thread_start_us":44945,"threads_started":1}
I20260619 14:50:22.037925 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.963888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:40459 (local address 127.0.241.3:44019)
0619 14:50:21.988751 (+ 24863us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:21.988853 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:21.994999 (+ 6146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:22.000344 (+ 5345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:22.000359 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:22.001105 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:22.002087 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.002117 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.008843 (+ 6726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.008851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.013585 (+ 4734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.013607 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.016553 (+ 2946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.016561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.016939 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.017897 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:22.017945 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:22.023163 (+ 5218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:22.032342 (+ 9179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:22.032351 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:22.032370 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:22.032912 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:22.033369 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:22.033374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:22.033377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:22.033608 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:22.034915 (+ 1307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:22.034928 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:22.035470 (+ 542us) client_negotiation.cc:769] Sending connection context
0619 14:50:22.035823 (+ 353us) client_negotiation.cc:240] Negotiation successful
0619 14:50:22.036431 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24322,"mutex_wait_us":16,"thread_start_us":34652,"threads_started":1}
I20260619 14:50:22.039902 8376 heartbeater.cc:347] Connected to a master server at 127.0.241.61:40459
I20260619 14:50:22.040233 8376 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:22.041152 8376 heartbeater.cc:511] Master 127.0.241.61:40459 requested a full tablet report, sending...
I20260619 14:50:22.043010 7721 ts_manager.cc:194] Registered new tserver with Master: 8cac0c45467f435a83c18d10cc73685a (127.0.241.3:44993)
I20260619 14:50:22.044018 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.964557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:33423 (local address 127.0.241.3:39281)
0619 14:50:21.998953 (+ 34396us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:21.998971 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:21.999082 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:22.000238 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:22.000251 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:22.008089 (+ 7838us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:22.008373 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.008384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.010879 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.010887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.013151 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.013169 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.017287 (+ 4118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.017295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.017512 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.021157 (+ 3645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:22.021179 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:22.025199 (+ 4020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:22.035486 (+ 10287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:22.035491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:22.035494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:22.036136 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:22.037855 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:22.037860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:22.037864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:22.038078 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:22.042260 (+ 4182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:22.042266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:22.042422 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 14:50:22.042579 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 14:50:22.042929 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34274,"mutex_wait_us":3,"thread_start_us":33528,"threads_started":1}
I20260619 14:50:22.045425 8379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.963516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:39281 (local address 127.0.241.60:33423)
0619 14:50:21.999405 (+ 35889us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:21.999410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:21.999423 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:50:21.999526 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:21.999531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:21.999643 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:21.999757 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:22.008549 (+ 8792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.009730 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.013339 (+ 3609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.013995 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.014109 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.026869 (+ 12760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:22.026876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:22.026881 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:22.026924 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:22.035339 (+ 8415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.037598 (+ 2259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.037603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.037607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.037750 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.041361 (+ 3611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.041368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.041373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.041704 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:22.041849 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.043000 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.043419 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35798,"thread_start_us":44593,"threads_started":1}
I20260619 14:50:22.045440 8380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.963621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:46777 (local address 127.0.241.62:33289)
0619 14:50:21.991204 (+ 27583us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:21.991215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:21.998505 (+ 7290us) server_negotiation.cc:407] Connection header received
0619 14:50:21.999734 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:21.999741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:21.999836 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:21.999972 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:22.011538 (+ 11566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.013038 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.015065 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.016038 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.016190 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.024485 (+ 8295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:22.024491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:22.024496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:22.024535 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:22.029137 (+ 4602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.037996 (+ 8859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.038005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.038010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.038196 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.039291 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.039296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.039304 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.039747 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:22.039893 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.044668 (+ 4775us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.044896 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27398,"thread_start_us":29507,"threads_started":1}
I20260619 14:50:22.045710 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:21.964055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.3:46777)
0619 14:50:21.998383 (+ 34328us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:21.998406 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:21.999567 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:22.000437 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:22.000444 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:22.009353 (+ 8909us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:22.009648 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.009659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.013189 (+ 3530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.013196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.014868 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.014887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.016530 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.016537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.017818 (+ 1281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.018967 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:22.018990 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:22.024264 (+ 5274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:22.035615 (+ 11351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:22.035619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:22.035622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:22.036805 (+ 1183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:22.038983 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:22.038987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:22.038992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:22.039168 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:22.044236 (+ 5068us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:22.044241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:22.044377 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 14:50:22.044538 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:50:22.044854 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34166,"mutex_wait_us":320,"spinlock_wait_cycles":21376,"thread_start_us":25615,"threads_started":1}
I20260619 14:50:22.046295 8374 heartbeater.cc:347] Connected to a master server at 127.0.241.60:33423
I20260619 14:50:22.046413 8374 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:22.046835 8374 heartbeater.cc:511] Master 127.0.241.60:33423 requested a full tablet report, sending...
I20260619 14:50:22.046964 8375 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33289
I20260619 14:50:22.047123 8375 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:22.047900 8375 heartbeater.cc:511] Master 127.0.241.62:33289 requested a full tablet report, sending...
I20260619 14:50:22.048054 7908 ts_manager.cc:194] Registered new tserver with Master: 8cac0c45467f435a83c18d10cc73685a (127.0.241.3:44993)
I20260619 14:50:22.048846 7123 ts_manager.cc:194] Re-registered known tserver with Master: 8cac0c45467f435a83c18d10cc73685a (127.0.241.3:44993)
I20260619 14:50:22.049836 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.000873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.0.241.62:33289)
0619 14:50:22.024071 (+ 23198us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.024078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.024096 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:50:22.024151 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.024158 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.024245 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.024344 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:22.028238 (+ 3894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.029525 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.031582 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.033026 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.033142 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.036719 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:22.036727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:22.036734 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:22.036773 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:22.041179 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.042034 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.042039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.042043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.044540 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.044979 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.044984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.044988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.045320 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:22.049054 (+ 3734us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.049187 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.049418 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23081,"thread_start_us":34604,"threads_started":1}
I20260619 14:50:22.050088 7123 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:46777
I20260619 14:50:22.050987 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.001365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.0.241.61:40459)
0619 14:50:22.016689 (+ 15324us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.016695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.023618 (+ 6923us) server_negotiation.cc:407] Connection header received
0619 14:50:22.024520 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.024528 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.024625 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.024755 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:22.029927 (+ 5172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.031234 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.036081 (+ 4847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.037009 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.037115 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.041990 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:22.041996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:22.042001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:22.042044 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:22.047581 (+ 5537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.048385 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.048390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.048394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.048540 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.048936 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.048943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.048946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.049271 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:22.049493 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.050408 (+ 915us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.050620 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15179,"thread_start_us":15056,"threads_started":1}
I20260619 14:50:22.053767 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.000884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.0.241.60:33423)
0619 14:50:22.025660 (+ 24776us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.025666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.025683 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:50:22.025734 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.025741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.025829 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.025954 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:22.028784 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.030068 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.031969 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.032588 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.032704 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.042029 (+ 9325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:22.042035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:22.042040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:22.042075 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:22.049584 (+ 7509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.051425 (+ 1841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.051466 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.051470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.051615 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.052056 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.052061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.052064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.052384 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:22.052610 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.053182 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.053396 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24646,"thread_start_us":24539,"threads_started":1}
I20260619 14:50:22.054504 7124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:22.061610 7124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:22.092484 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.080367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:38131 (local address 127.0.0.1:48958)
0619 14:50:22.081892 (+ 1525us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:22.081910 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:22.082040 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:22.083204 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:22.083211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:22.083247 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:22.083715 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.083731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.086014 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.086023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.088982 (+ 2959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.088997 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.091328 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.091338 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.091631 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.091682 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 14:50:22.091763 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:50:22.091954 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"thread_start_us":1269,"threads_started":1}
I20260619 14:50:22.094069 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.080606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.0.241.1:38131)
0619 14:50:22.082512 (+ 1906us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.082522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.082549 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:50:22.082638 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.082643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.082936 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.083952 (+ 1016us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:22.084219 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.085824 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.089198 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.091071 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.093215 (+ 2144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.093481 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.093567 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.093734 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1739,"thread_start_us":1737,"threads_started":1}
I20260619 14:50:22.099886 8022 tablet_service.cc:1467] Processing CreateTablet for tablet 41631f8db77b4c2eb79bcb01f9748fe4 (DEFAULT_TABLE table=test-table [id=f0c8c273f71842a7958fa845e8bfe78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:22.101245 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41631f8db77b4c2eb79bcb01f9748fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:22.103842 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.080868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:44993 (local address 127.0.0.1:45272)
0619 14:50:22.092918 (+ 12050us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:22.092942 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:22.093039 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:22.095327 (+ 2288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:22.095335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:22.095372 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:22.095639 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.095653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.098208 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.098213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.100018 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.100032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.102748 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.102754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.103070 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.103137 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 14:50:22.103244 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:50:22.103377 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11921,"mutex_wait_us":22,"thread_start_us":16356,"threads_started":1}
I20260619 14:50:22.103861 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.081134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.0.241.3:44993)
0619 14:50:22.094594 (+ 13460us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.094606 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.094645 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:50:22.094749 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.094759 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.095090 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.095314 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:22.096391 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.098065 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.101166 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.102590 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.102702 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.102910 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.103318 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.103534 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12995,"thread_start_us":13074,"threads_started":1}
I20260619 14:50:22.105257 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.083173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.0.241.2:38485)
0619 14:50:22.096394 (+ 13221us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.096402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.096440 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:50:22.096524 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.096531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.096750 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.096923 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:22.098068 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.098974 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.101114 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.102572 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.102700 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.102957 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.104112 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.104264 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13078,"thread_start_us":13338,"threads_started":1}
I20260619 14:50:22.103928 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.082887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:38485 (local address 127.0.0.1:56688)
0619 14:50:22.096087 (+ 13200us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:22.096109 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:22.096218 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:22.097484 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:22.097488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:22.097513 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:22.097858 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.097870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.099110 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.099118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.100982 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.100993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.103056 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.103083 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.103303 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.103339 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:50:22.103431 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:50:22.103567 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13057,"thread_start_us":12886,"threads_started":1}
I20260619 14:50:22.108088 8309 tablet_service.cc:1467] Processing CreateTablet for tablet 41631f8db77b4c2eb79bcb01f9748fe4 (DEFAULT_TABLE table=test-table [id=f0c8c273f71842a7958fa845e8bfe78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:22.109253 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41631f8db77b4c2eb79bcb01f9748fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:22.108853 8168 tablet_service.cc:1467] Processing CreateTablet for tablet 41631f8db77b4c2eb79bcb01f9748fe4 (DEFAULT_TABLE table=test-table [id=f0c8c273f71842a7958fa845e8bfe78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:22.110073 8168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41631f8db77b4c2eb79bcb01f9748fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:22.117264 8411 tablet_bootstrap.cc:492] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b: Bootstrap starting.
I20260619 14:50:22.120900 8411 tablet_bootstrap.cc:654] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:22.122346 8411 log.cc:824] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:22.128350 8412 tablet_bootstrap.cc:492] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a: Bootstrap starting.
I20260619 14:50:22.132468 8414 tablet_bootstrap.cc:492] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f: Bootstrap starting.
I20260619 14:50:22.132870 8412 tablet_bootstrap.cc:654] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:22.133880 8412 log.cc:824] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:22.135519 8414 tablet_bootstrap.cc:654] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:22.136418 8414 log.cc:824] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:22.136647 8412 tablet_bootstrap.cc:492] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a: No bootstrap required, opened a new log
I20260619 14:50:22.136919 8412 ts_tablet_manager.cc:1397] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 14:50:22.138208 8411 tablet_bootstrap.cc:492] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b: No bootstrap required, opened a new log
I20260619 14:50:22.138408 8411 ts_tablet_manager.cc:1397] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.008s
I20260619 14:50:22.141099 8414 tablet_bootstrap.cc:492] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f: No bootstrap required, opened a new log
I20260619 14:50:22.141337 8414 ts_tablet_manager.cc:1397] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260619 14:50:22.145260 8412 raft_consensus.cc:348] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:22.145604 8412 raft_consensus.cc:374] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:22.145707 8412 raft_consensus.cc:729] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cac0c45467f435a83c18d10cc73685a, State: Initialized, Role: FOLLOWER
I20260619 14:50:22.146337 8412 consensus_queue.cc:260] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:22.147480 8375 heartbeater.cc:503] Master 127.0.241.62:33289 was elected leader, sending a full tablet report...
I20260619 14:50:22.150111 8411 raft_consensus.cc:348] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:22.150501 8411 raft_consensus.cc:374] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:22.150583 8411 raft_consensus.cc:729] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b840e7d2eb14f1e9cc4094eecbe287b, State: Initialized, Role: FOLLOWER
I20260619 14:50:22.151122 8411 consensus_queue.cc:260] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:22.151835 8414 raft_consensus.cc:348] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:22.152298 8414 raft_consensus.cc:374] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:22.152408 8414 raft_consensus.cc:729] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e46bb4f46e4d309506a4167f43343f, State: Initialized, Role: FOLLOWER
I20260619 14:50:22.153070 8414 consensus_queue.cc:260] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:22.153537 8412 ts_tablet_manager.cc:1428] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a: Time spent starting tablet: real 0.017s user 0.011s sys 0.000s
I20260619 14:50:22.160974 8411 ts_tablet_manager.cc:1428] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b: Time spent starting tablet: real 0.022s user 0.010s sys 0.010s
I20260619 14:50:22.162601 8414 ts_tablet_manager.cc:1428] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f: Time spent starting tablet: real 0.021s user 0.020s sys 0.000s
I20260619 14:50:22.162878 8234 heartbeater.cc:503] Master 127.0.241.62:33289 was elected leader, sending a full tablet report...
W20260619 14:50:22.246641 8377 tablet.cc:2369] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:50:22.273339 8090 tablet.cc:2369] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:50:22.383628 8236 tablet.cc:2369] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:22.419421 8418 raft_consensus.cc:482] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:22.419847 8418 raft_consensus.cc:504] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:22.421610 8418 leader_election.cc:288] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cac0c45467f435a83c18d10cc73685a (127.0.241.3:44993), a6e46bb4f46e4d309506a4167f43343f (127.0.241.2:38485)
I20260619 14:50:22.431502 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.422094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:43065 (local address 127.0.241.2:38485)
0619 14:50:22.422309 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.422318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.422896 (+ 578us) server_negotiation.cc:407] Connection header received
0619 14:50:22.423042 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.423048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.423123 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.423224 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:22.423992 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.425076 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.428594 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.429824 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.429946 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.430045 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.431088 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.431213 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:50:22.431502 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.422094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:48989 (local address 127.0.241.3:44993)
0619 14:50:22.422291 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.422300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.422846 (+ 546us) server_negotiation.cc:407] Connection header received
0619 14:50:22.423043 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.423048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.423123 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.423224 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:22.423989 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.425076 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.428604 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.429812 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.429946 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.430050 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.431088 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.431212 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:50:22.431934 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.422042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:38485 (local address 127.0.241.1:43065)
0619 14:50:22.422795 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:22.422810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:22.422909 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:22.423255 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:22.423260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:22.423285 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:22.423816 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.423834 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.425256 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.425264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.428437 (+ 3173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.428454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.430132 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.430141 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.430865 (+ 724us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.430950 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 14:50:22.431054 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 14:50:22.431301 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":405,"spinlock_wait_cycles":896,"thread_start_us":491,"threads_started":1}
I20260619 14:50:22.431967 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.421885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:44993 (local address 127.0.241.1:48989)
0619 14:50:22.422716 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:22.422737 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:22.422906 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:22.423255 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:22.423294 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:22.423314 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:22.423839 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.423848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.425256 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.425264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.428437 (+ 3173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:22.428454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.430154 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.430159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:22.430792 (+ 633us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.430942 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 14:50:22.431054 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:50:22.431301 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":466,"thread_start_us":425,"threads_started":1}
I20260619 14:50:22.432884 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41631f8db77b4c2eb79bcb01f9748fe4" candidate_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cac0c45467f435a83c18d10cc73685a" is_pre_election: true
I20260619 14:50:22.432911 8188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41631f8db77b4c2eb79bcb01f9748fe4" candidate_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e46bb4f46e4d309506a4167f43343f" is_pre_election: true
I20260619 14:50:22.433332 8329 raft_consensus.cc:2454] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b840e7d2eb14f1e9cc4094eecbe287b in term 0.
I20260619 14:50:22.433385 8188 raft_consensus.cc:2454] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b840e7d2eb14f1e9cc4094eecbe287b in term 0.
I20260619 14:50:22.433955 7976 leader_election.cc:302] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b840e7d2eb14f1e9cc4094eecbe287b, 8cac0c45467f435a83c18d10cc73685a; no voters:
I20260619 14:50:22.434324 8418 raft_consensus.cc:2781] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:22.434484 8418 raft_consensus.cc:482] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:22.434608 8418 raft_consensus.cc:3037] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:22.436977 8418 raft_consensus.cc:504] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:22.437700 8418 leader_election.cc:288] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [CANDIDATE]: Term 1 election: Requested vote from peers 8cac0c45467f435a83c18d10cc73685a (127.0.241.3:44993), a6e46bb4f46e4d309506a4167f43343f (127.0.241.2:38485)
I20260619 14:50:22.438133 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41631f8db77b4c2eb79bcb01f9748fe4" candidate_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cac0c45467f435a83c18d10cc73685a"
I20260619 14:50:22.438141 8188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41631f8db77b4c2eb79bcb01f9748fe4" candidate_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e46bb4f46e4d309506a4167f43343f"
I20260619 14:50:22.438354 8329 raft_consensus.cc:3037] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:22.438354 8188 raft_consensus.cc:3037] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:22.441687 8329 raft_consensus.cc:2454] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b840e7d2eb14f1e9cc4094eecbe287b in term 1.
I20260619 14:50:22.441679 8188 raft_consensus.cc:2454] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b840e7d2eb14f1e9cc4094eecbe287b in term 1.
I20260619 14:50:22.442094 7977 leader_election.cc:302] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b840e7d2eb14f1e9cc4094eecbe287b, a6e46bb4f46e4d309506a4167f43343f; no voters:
I20260619 14:50:22.442415 8418 raft_consensus.cc:2781] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:22.442883 8418 raft_consensus.cc:686] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 1 LEADER]: Becoming Leader. State: Replica: 6b840e7d2eb14f1e9cc4094eecbe287b, State: Running, Role: LEADER
I20260619 14:50:22.443490 8418 consensus_queue.cc:237] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:22.449337 7123 catalog_manager.cc:5495] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b840e7d2eb14f1e9cc4094eecbe287b (127.0.241.1). New cstate: current_term: 1 leader_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } health_report { overall_health: UNKNOWN } } }
I20260619 14:50:22.541502 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.535704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.0.241.1:38131)
0619 14:50:22.535959 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.535966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.536083 (+ 117us) server_negotiation.cc:407] Connection header received
0619 14:50:22.536351 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.536356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.536435 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.536606 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:22.537030 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.537777 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.539193 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.539686 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.539829 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.540341 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:22.540859 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:22.541031 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.541141 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.541225 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 14:50:22.549755 8188 raft_consensus.cc:1261] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f [term 1 FOLLOWER]: Refusing update from remote peer 6b840e7d2eb14f1e9cc4094eecbe287b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:22.549964 8329 raft_consensus.cc:1261] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a [term 1 FOLLOWER]: Refusing update from remote peer 6b840e7d2eb14f1e9cc4094eecbe287b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:22.550861 8422 consensus_queue.cc:1035] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:22.551270 8418 consensus_queue.cc:1035] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:22.564531 8427 mvcc.cc:205] Tried to move back new op lower bound from 7298583029954437120 to 7298583029532323840. Current Snapshot: MvccSnapshot[applied={T|T < 7298583029954437120}]
I20260619 14:50:22.579807 7124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:22.580135 7124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:22.593364 7124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4002d8e4b9404f0c9d8901c3203d3483: Sending DeleteTablet for 3 replicas of tablet 41631f8db77b4c2eb79bcb01f9748fe4
I20260619 14:50:22.594732 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet 41631f8db77b4c2eb79bcb01f9748fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:22 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:45272
I20260619 14:50:22.594856 8168 tablet_service.cc:1514] Processing DeleteTablet for tablet 41631f8db77b4c2eb79bcb01f9748fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:22 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:56688
I20260619 14:50:22.621729 8022 tablet_service.cc:1514] Processing DeleteTablet for tablet 41631f8db77b4c2eb79bcb01f9748fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:22 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:48958
I20260619 14:50:22.647960 8439 tablet_replica.cc:330] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b: stopping tablet replica
I20260619 14:50:22.648646 8439 raft_consensus.cc:2229] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:22.649501 8439 raft_consensus.cc:2258] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:22.652128 8439 ts_tablet_manager.cc:1905] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:22.655345 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 7092
I20260619 14:50:22.656742 7154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:22.663447 8439 ts_tablet_manager.cc:1918] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:22.671245 8439 log.cc:1196] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/41631f8db77b4c2eb79bcb01f9748fe4
I20260619 14:50:22.672065 8439 ts_tablet_manager.cc:1939] T 41631f8db77b4c2eb79bcb01f9748fe4 P 6b840e7d2eb14f1e9cc4094eecbe287b: Deleting consensus metadata
I20260619 14:50:22.676254 8436 tablet_replica.cc:330] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f: stopping tablet replica
I20260619 14:50:22.676561 8435 tablet_replica.cc:330] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a: stopping tablet replica
I20260619 14:50:22.676738 8436 raft_consensus.cc:2229] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:22.676985 8435 raft_consensus.cc:2229] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:22.677230 8436 raft_consensus.cc:2258] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:22.677415 8435 raft_consensus.cc:2258] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:22.679744 8435 ts_tablet_manager.cc:1905] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:22.687202 8435 ts_tablet_manager.cc:1918] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:22.687486 8435 log.cc:1196] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/41631f8db77b4c2eb79bcb01f9748fe4
I20260619 14:50:22.688177 8435 ts_tablet_manager.cc:1939] T 41631f8db77b4c2eb79bcb01f9748fe4 P 8cac0c45467f435a83c18d10cc73685a: Deleting consensus metadata
I20260619 14:50:22.689376 8436 ts_tablet_manager.cc:1905] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:22.697556 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.603316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.0.241.61:40459)
0619 14:50:22.639602 (+ 36286us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.639610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.639637 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:50:22.639730 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.639738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.639849 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.639985 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:22.653025 (+ 13040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.654555 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.677589 (+ 23034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.678663 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.683906 (+ 5243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.684269 (+ 363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:22.684279 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:22.684287 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:22.684334 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:22.693343 (+ 9009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.694288 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.694294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.694298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.694455 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.694915 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.694919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.694923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.695291 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:22.696819 (+ 1528us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.696937 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.697158 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36139,"thread_start_us":44976,"threads_started":1}
I20260619 14:50:22.700459 8436 ts_tablet_manager.cc:1918] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:22.700675 8436 log.cc:1196] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/41631f8db77b4c2eb79bcb01f9748fe4
I20260619 14:50:22.701246 8436 ts_tablet_manager.cc:1939] T 41631f8db77b4c2eb79bcb01f9748fe4 P a6e46bb4f46e4d309506a4167f43343f: Deleting consensus metadata
I20260619 14:50:22.715806 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.640920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.0.241.60:33423)
0619 14:50:22.670192 (+ 29272us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.670203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.696517 (+ 26314us) server_negotiation.cc:407] Connection header received
0619 14:50:22.696711 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.696718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.696811 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.696927 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:22.700186 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.701363 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.703729 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.704498 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.704633 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.708887 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:22.708894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:22.708901 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:22.708941 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:22.712762 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.713577 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.713583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.713587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.713734 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.714196 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.714201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.714205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.714555 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:22.714800 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.715193 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.715417 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29056,"thread_start_us":29348,"threads_started":1}
I20260619 14:50:22.819401 7109 catalog_manager.cc:4841] TS 8cac0c45467f435a83c18d10cc73685a (127.0.241.3:44993): tablet 41631f8db77b4c2eb79bcb01f9748fe4 (table test-table [id=f0c8c273f71842a7958fa845e8bfe78e]) successfully deleted
I20260619 14:50:22.819859 7110 catalog_manager.cc:4841] TS 6b840e7d2eb14f1e9cc4094eecbe287b (127.0.241.1:38131): tablet 41631f8db77b4c2eb79bcb01f9748fe4 (table test-table [id=f0c8c273f71842a7958fa845e8bfe78e]) successfully deleted
W20260619 14:50:22.820067 7110 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.241.1:38131, user_credentials={real_user=slave}} blocked reactor thread for 141938us
I20260619 14:50:22.820302 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 7092
I20260619 14:50:22.880865 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.846673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.0.241.61:40459)
0619 14:50:22.849081 (+ 2408us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.849091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.857283 (+ 8192us) server_negotiation.cc:407] Connection header received
0619 14:50:22.857856 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.857867 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.858052 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.858156 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:22.861499 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.863059 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.867205 (+ 4146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.867977 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.868099 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.874375 (+ 6276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:22.874385 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:22.874392 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:22.874442 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:22.877713 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.878502 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.878507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.878510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.878642 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.879099 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.879104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.879107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.879490 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:22.879709 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.880272 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.880510 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2181}
I20260619 14:50:22.882850 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:22.846419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.0.241.60:33423)
0619 14:50:22.851305 (+ 4886us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:22.851315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:22.866759 (+ 15444us) server_negotiation.cc:407] Connection header received
0619 14:50:22.867094 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:22.867100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:22.867193 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:22.867266 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:22.869563 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.870645 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.872311 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:22.872744 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:22.872895 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:22.876009 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:22.876014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:22.876019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:22.876053 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:22.879123 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.879891 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.879896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.879898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.880017 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:22.880429 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:22.880440 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:22.880442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:22.880705 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:22.880959 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:22.881696 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 14:50:22.882543 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4680}
I20260619 14:50:23.573302 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:23.571807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.2:39233)
0619 14:50:23.572945 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:23.573090 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1021,"thread_start_us":872,"threads_started":1}
W20260619 14:50:23.574309 8234 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:33289 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
I20260619 14:50:23.601164 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:23.600457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.3:34225)
0619 14:50:23.600895 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:23.601003 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":212,"threads_started":1}
W20260619 14:50:23.601564 8375 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:33289 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
I20260619 14:50:23.622581 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:23.621808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.1:49001)
0619 14:50:23.622283 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:23.622406 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":279,"threads_started":1}
W20260619 14:50:23.623036 8088 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:33289 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
I20260619 14:50:24.098544 8452 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4002d8e4b9404f0c9d8901c3203d3483)
I20260619 14:50:24.098834 8452 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } }
I20260619 14:50:24.100234 8452 leader_election.cc:288] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4002d8e4b9404f0c9d8901c3203d3483 (127.0.241.62:33289), 04fac3b86b9f46cda4436820eb3e55fb (127.0.241.61:40459)
I20260619 14:50:24.101464 8456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4002d8e4b9404f0c9d8901c3203d3483)
I20260619 14:50:24.101749 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.100409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:56952)
0619 14:50:24.101207 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.101429 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":485,"threads_started":1}
W20260619 14:50:24.102064 7895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
I20260619 14:50:24.101788 8456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } }
I20260619 14:50:24.103978 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.103362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:56954)
0619 14:50:24.103667 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.103834 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
W20260619 14:50:24.104148 7895 leader_election.cc:334] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4002d8e4b9404f0c9d8901c3203d3483 (127.0.241.62:33289): Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
I20260619 14:50:24.108175 8456 leader_election.cc:288] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4002d8e4b9404f0c9d8901c3203d3483 (127.0.241.62:33289), cd5adced693746f4b6b64150310bc02d (127.0.241.60:33423)
I20260619 14:50:24.109566 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.100447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:40459 (local address 127.0.0.1:42740)
0619 14:50:24.101950 (+ 1503us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.101967 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:24.102107 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:24.103127 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:24.103131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:24.103183 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:24.103459 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.103474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.105055 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.105061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.106939 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.106954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.108772 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.108778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.108990 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.109027 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 14:50:24.109119 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:50:24.109267 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1411,"thread_start_us":3027,"threads_started":1}
I20260619 14:50:24.110052 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.100570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.0.241.61:40459)
0619 14:50:24.102699 (+ 2129us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:24.102704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:24.102722 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:50:24.102793 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:24.102798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:24.102866 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:24.102958 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:24.103764 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.104903 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.107107 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.108439 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.108578 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.108687 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:24.109559 (+ 872us) server_negotiation.cc:299] Negotiation successful
0619 14:50:24.109703 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1963,"thread_start_us":1739,"threads_started":1}
I20260619 14:50:24.110077 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.105711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:56960)
0619 14:50:24.109570 (+ 3859us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.109764 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3750,"thread_start_us":3234,"threads_started":1}
W20260619 14:50:24.110474 7708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
I20260619 14:50:24.110920 7730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd5adced693746f4b6b64150310bc02d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04fac3b86b9f46cda4436820eb3e55fb" is_pre_election: true
I20260619 14:50:24.111325 7730 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd5adced693746f4b6b64150310bc02d in term 1.
I20260619 14:50:24.111958 7892 leader_election.cc:302] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04fac3b86b9f46cda4436820eb3e55fb, cd5adced693746f4b6b64150310bc02d; no voters: 4002d8e4b9404f0c9d8901c3203d3483
I20260619 14:50:24.112353 8452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 14:50:24.112465 8452 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4002d8e4b9404f0c9d8901c3203d3483)
I20260619 14:50:24.112546 8452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:50:24.113047 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.112688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:56962)
0619 14:50:24.112832 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.112917 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:50:24.113292 7708 leader_election.cc:334] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4002d8e4b9404f0c9d8901c3203d3483 (127.0.241.62:33289): Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
I20260619 14:50:24.115379 8452 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } }
I20260619 14:50:24.116286 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.105750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.0.241.60:33423)
0619 14:50:24.108786 (+ 3036us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:24.108791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:24.109403 (+ 612us) server_negotiation.cc:407] Connection header received
0619 14:50:24.109457 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:24.109461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:24.109521 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:24.109614 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:24.110695 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.111669 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.114335 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.115262 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.115405 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.115529 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:24.115802 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 14:50:24.115926 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2902,"thread_start_us":2793,"threads_started":1}
I20260619 14:50:24.116322 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.105330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:33423 (local address 127.0.0.1:41154)
0619 14:50:24.109176 (+ 3846us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.109193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:24.109301 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:24.110017 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:24.110023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:24.110060 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:24.110489 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.110500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.111855 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.111860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.114165 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.114179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.115426 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.115454 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.115617 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.115669 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 14:50:24.115782 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 14:50:24.115904 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3719,"thread_start_us":4007,"threads_started":1}
I20260619 14:50:24.116434 8452 leader_election.cc:288] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [CANDIDATE]: Term 2 election: Requested vote from peers 4002d8e4b9404f0c9d8901c3203d3483 (127.0.241.62:33289), 04fac3b86b9f46cda4436820eb3e55fb (127.0.241.61:40459)
I20260619 14:50:24.116557 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.116137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:56964)
0619 14:50:24.116281 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.116386 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:50:24.117009 7730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd5adced693746f4b6b64150310bc02d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04fac3b86b9f46cda4436820eb3e55fb"
I20260619 14:50:24.117216 7730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:50:24.117185 7918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04fac3b86b9f46cda4436820eb3e55fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cd5adced693746f4b6b64150310bc02d" is_pre_election: true
I20260619 14:50:24.117542 7918 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 04fac3b86b9f46cda4436820eb3e55fb in current term 2: Already voted for candidate cd5adced693746f4b6b64150310bc02d in this term.
I20260619 14:50:24.118070 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.117676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:56978)
0619 14:50:24.117844 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.117959 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 14:50:24.118098 7706 leader_election.cc:302] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04fac3b86b9f46cda4436820eb3e55fb; no voters: 4002d8e4b9404f0c9d8901c3203d3483, cd5adced693746f4b6b64150310bc02d
W20260619 14:50:24.118637 7895 leader_election.cc:334] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4002d8e4b9404f0c9d8901c3203d3483 (127.0.241.62:33289): Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
I20260619 14:50:24.120153 7730 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd5adced693746f4b6b64150310bc02d in term 2.
I20260619 14:50:24.120347 8456 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:50:24.120635 7892 leader_election.cc:302] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04fac3b86b9f46cda4436820eb3e55fb, cd5adced693746f4b6b64150310bc02d; no voters: 4002d8e4b9404f0c9d8901c3203d3483
I20260619 14:50:24.120945 8452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:50:24.121435 8452 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [term 2 LEADER]: Becoming Leader. State: Replica: cd5adced693746f4b6b64150310bc02d, State: Running, Role: LEADER
I20260619 14:50:24.121836 8452 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } }
I20260619 14:50:24.123849 8462 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd5adced693746f4b6b64150310bc02d. Latest consensus state: current_term: 2 leader_uuid: "cd5adced693746f4b6b64150310bc02d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } } }
I20260619 14:50:24.124264 8462 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:24.124902 8464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:24.128342 8464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:24.129128 8464 catalog_manager.cc:1259] Loaded cluster ID: d1e6b37ed3e14281bc735000d4ab54a6
I20260619 14:50:24.129195 8464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:24.130273 8464 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:24.131036 8464 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: Loaded TSK: 0
I20260619 14:50:24.132018 8464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:50:24.161840 7907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:24.168678 7907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:24.172801 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.172413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:57002)
0619 14:50:24.172586 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.172667 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:50:24.172847 7730 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [term 2 FOLLOWER]: Refusing update from remote peer cd5adced693746f4b6b64150310bc02d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 14:50:24.173692 8462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [LEADER]: Connected to new peer: Peer: permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 14:50:24.173838 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.173483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:57018)
0619 14:50:24.173609 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.173684 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:50:24.174041 7895 consensus_peers.cc:487] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d -> Peer 4002d8e4b9404f0c9d8901c3203d3483 (127.0.241.62:33289): Couldn't send request to peer 4002d8e4b9404f0c9d8901c3203d3483. Status: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:50:24.176497 8456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd5adced693746f4b6b64150310bc02d. Latest consensus state: current_term: 2 leader_uuid: "cd5adced693746f4b6b64150310bc02d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } } }
I20260619 14:50:24.176766 8456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:50:24.178115 8462 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "cd5adced693746f4b6b64150310bc02d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } } }
I20260619 14:50:24.178402 8462 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:24.178725 8466 mvcc.cc:205] Tried to move back new op lower bound from 7298583036605173760 to 7298583036406571008. Current Snapshot: MvccSnapshot[applied={T|T < 7298583036605173760}]
I20260619 14:50:24.178640 8456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "cd5adced693746f4b6b64150310bc02d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4002d8e4b9404f0c9d8901c3203d3483" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33289 } } peers { permanent_uuid: "04fac3b86b9f46cda4436820eb3e55fb" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 40459 } attrs { promote: false } } peers { permanent_uuid: "cd5adced693746f4b6b64150310bc02d" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 33423 } attrs { promote: false } } }
I20260619 14:50:24.178885 8456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fac3b86b9f46cda4436820eb3e55fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:50:24.196982 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.187375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:44993 (local address 127.0.0.1:45280)
0619 14:50:24.187547 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.187563 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:24.187650 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:24.188745 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:24.188751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:24.188784 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:24.189095 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.189107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.191048 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.191056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.193941 (+ 2885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.193960 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.196131 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.196138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.196327 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.196371 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 14:50:24.196441 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:50:24.196582 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":26}
I20260619 14:50:24.197242 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.186556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:38131 (local address 127.0.0.1:48974)
0619 14:50:24.186725 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.186740 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:24.186824 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:24.187539 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:24.187544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:24.187570 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:24.187829 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.187841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.189781 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.189789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.192516 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.192528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.196153 (+ 3625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.196162 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.196460 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.196515 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 14:50:24.196595 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:50:24.196808 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":39}
I20260619 14:50:24.198498 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.186734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.0.241.1:38131)
0619 14:50:24.187191 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:24.187198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:24.187216 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:50:24.187284 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:24.187290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:24.187361 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:24.187974 (+ 613us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:24.188171 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.189561 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.194204 (+ 4643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.195896 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.197655 (+ 1759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.197837 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:24.197905 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 14:50:24.198100 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":235,"threads_started":1}
I20260619 14:50:24.198611 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.187491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.0.241.3:44993)
0619 14:50:24.188003 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:24.188011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:24.188030 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:50:24.188106 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:24.188114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:24.188198 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:24.189251 (+ 1053us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:24.189436 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.190860 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.194166 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.195850 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.197807 (+ 1957us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.197959 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:24.198018 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 14:50:24.198219 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":304,"threads_started":1}
I20260619 14:50:24.199568 8309 tablet_service.cc:1467] Processing CreateTablet for tablet aefcbcd67ed94c059b3197850754b798 (DEFAULT_TABLE table=test-table [id=cf3f11e2cb584fc1854c55ad217c9ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:24.200169 8309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefcbcd67ed94c059b3197850754b798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:24.201246 8022 tablet_service.cc:1467] Processing CreateTablet for tablet aefcbcd67ed94c059b3197850754b798 (DEFAULT_TABLE table=test-table [id=cf3f11e2cb584fc1854c55ad217c9ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:24.201663 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefcbcd67ed94c059b3197850754b798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:24.204944 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.188237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.0.241.2:38485)
0619 14:50:24.199869 (+ 11632us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:24.199876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:24.199894 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:50:24.199960 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:24.199963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:24.200016 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:24.200130 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:24.200503 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.201544 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.203232 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.204050 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.204189 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.204286 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:24.204543 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 14:50:24.204672 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11483,"thread_start_us":11378,"threads_started":1}
I20260619 14:50:24.204978 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.188071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:38485 (local address 127.0.0.1:56700)
0619 14:50:24.197303 (+ 9232us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.197327 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:24.197460 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:24.200143 (+ 2683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:24.200147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:24.200170 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:24.200362 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.200372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.201723 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.201727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.203050 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.203061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.204210 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.204216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.204388 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.204410 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:50:24.204519 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:50:24.204646 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9087,"thread_start_us":12349,"threads_started":1}
I20260619 14:50:24.205785 8168 tablet_service.cc:1467] Processing CreateTablet for tablet aefcbcd67ed94c059b3197850754b798 (DEFAULT_TABLE table=test-table [id=cf3f11e2cb584fc1854c55ad217c9ec4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:24.206197 8168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aefcbcd67ed94c059b3197850754b798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:24.208815 8476 tablet_bootstrap.cc:492] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b: Bootstrap starting.
I20260619 14:50:24.209192 8475 tablet_bootstrap.cc:492] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a: Bootstrap starting.
I20260619 14:50:24.210579 8475 tablet_bootstrap.cc:654] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:24.210587 8476 tablet_bootstrap.cc:654] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:24.212575 8479 tablet_bootstrap.cc:492] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f: Bootstrap starting.
I20260619 14:50:24.213259 8476 tablet_bootstrap.cc:492] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b: No bootstrap required, opened a new log
I20260619 14:50:24.213449 8476 ts_tablet_manager.cc:1397] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 14:50:24.213991 8479 tablet_bootstrap.cc:654] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:24.214483 8476 raft_consensus.cc:348] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:24.214967 8476 raft_consensus.cc:374] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:24.215131 8476 raft_consensus.cc:729] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b840e7d2eb14f1e9cc4094eecbe287b, State: Initialized, Role: FOLLOWER
I20260619 14:50:24.215528 8476 consensus_queue.cc:260] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:24.216598 8476 ts_tablet_manager.cc:1428] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:24.218506 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.217342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.1:39289)
0619 14:50:24.217988 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.218175 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"threads_started":1}
I20260619 14:50:24.219002 8475 tablet_bootstrap.cc:492] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a: No bootstrap required, opened a new log
I20260619 14:50:24.219174 8475 ts_tablet_manager.cc:1397] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 14:50:24.220005 8475 raft_consensus.cc:348] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:24.220264 8475 raft_consensus.cc:374] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:24.220423 8475 raft_consensus.cc:729] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cac0c45467f435a83c18d10cc73685a, State: Initialized, Role: FOLLOWER
I20260619 14:50:24.220862 8087 heartbeater.cc:503] Master 127.0.241.60:33423 was elected leader, sending a full tablet report...
I20260619 14:50:24.220832 8475 consensus_queue.cc:260] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:24.224792 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.222434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.1:50863)
0619 14:50:24.224456 (+ 2022us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.224571 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1878}
I20260619 14:50:24.225941 8475 ts_tablet_manager.cc:1428] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260619 14:50:24.226619 8479 tablet_bootstrap.cc:492] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f: No bootstrap required, opened a new log
I20260619 14:50:24.226765 8479 ts_tablet_manager.cc:1397] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260619 14:50:24.227550 8374 heartbeater.cc:503] Master 127.0.241.60:33423 was elected leader, sending a full tablet report...
I20260619 14:50:24.227597 8479 raft_consensus.cc:348] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:24.227818 8479 raft_consensus.cc:374] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:24.227905 8479 raft_consensus.cc:729] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e46bb4f46e4d309506a4167f43343f, State: Initialized, Role: FOLLOWER
I20260619 14:50:24.228279 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.226700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.3:57881)
0619 14:50:24.227866 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.228020 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1029,"threads_started":1}
I20260619 14:50:24.228117 8479 consensus_queue.cc:260] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:24.228947 8479 ts_tablet_manager.cc:1428] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 14:50:24.230769 8233 heartbeater.cc:503] Master 127.0.241.60:33423 was elected leader, sending a full tablet report...
I20260619 14:50:24.231308 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.230916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.3:45707)
0619 14:50:24.231058 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.231164 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:50:24.231307 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.229363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.2:39319)
0619 14:50:24.230368 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.230526 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":274,"threads_started":1}
I20260619 14:50:24.232173 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.231867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.2:43919)
0619 14:50:24.232002 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.232077 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 14:50:24.351974 8485 raft_consensus.cc:482] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:24.352213 8485 raft_consensus.cc:504] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:24.353741 8485 leader_election.cc:288] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b840e7d2eb14f1e9cc4094eecbe287b (127.0.241.1:38131), 8cac0c45467f435a83c18d10cc73685a (127.0.241.3:44993)
I20260619 14:50:24.360049 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.354074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:44993 (local address 127.0.241.2:36793)
0619 14:50:24.354391 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.354415 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:24.354639 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:24.355070 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:24.355077 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:24.355103 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:24.355542 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.355556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.356644 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.356649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.358119 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.358130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.359233 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.359242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.359510 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.359546 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:50:24.359619 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:50:24.359748 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":27}
I20260619 14:50:24.360074 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.354240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:36793 (local address 127.0.241.3:44993)
0619 14:50:24.354434 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:24.354440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:24.354502 (+ 62us) server_negotiation.cc:407] Connection header received
0619 14:50:24.354823 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:24.354829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:24.354930 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:24.355028 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:24.355689 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.356513 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.358267 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.359037 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.359130 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.359205 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:24.359667 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 14:50:24.359799 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:50:24.360152 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.354157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:35759 (local address 127.0.241.1:38131)
0619 14:50:24.354352 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:24.354359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:24.354954 (+ 595us) server_negotiation.cc:407] Connection header received
0619 14:50:24.355124 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:24.355129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:24.355225 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:24.355321 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:24.355974 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.356718 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.358396 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.359175 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.359300 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.359391 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:24.359785 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 14:50:24.359913 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:50:24.360621 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.354148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:38131 (local address 127.0.241.2:35759)
0619 14:50:24.354831 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.354847 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:24.354981 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:24.355345 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:24.355348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:24.355367 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:24.355807 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.355819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.356855 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.356858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.358244 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:24.358257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.359398 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.359402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:24.359592 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.359645 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 14:50:24.359737 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:50:24.359849 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":26,"thread_start_us":391,"threads_started":1}
I20260619 14:50:24.360863 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefcbcd67ed94c059b3197850754b798" candidate_uuid: "a6e46bb4f46e4d309506a4167f43343f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cac0c45467f435a83c18d10cc73685a" is_pre_election: true
I20260619 14:50:24.361135 8329 raft_consensus.cc:2454] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e46bb4f46e4d309506a4167f43343f in term 0.
I20260619 14:50:24.361505 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefcbcd67ed94c059b3197850754b798" candidate_uuid: "a6e46bb4f46e4d309506a4167f43343f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" is_pre_election: true
I20260619 14:50:24.361621 8122 leader_election.cc:302] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cac0c45467f435a83c18d10cc73685a, a6e46bb4f46e4d309506a4167f43343f; no voters:
I20260619 14:50:24.361888 8042 raft_consensus.cc:2454] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e46bb4f46e4d309506a4167f43343f in term 0.
I20260619 14:50:24.362020 8485 raft_consensus.cc:2781] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:24.362128 8485 raft_consensus.cc:482] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:24.362215 8485 raft_consensus.cc:3037] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:24.364440 8485 raft_consensus.cc:504] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:24.365053 8485 leader_election.cc:288] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [CANDIDATE]: Term 1 election: Requested vote from peers 6b840e7d2eb14f1e9cc4094eecbe287b (127.0.241.1:38131), 8cac0c45467f435a83c18d10cc73685a (127.0.241.3:44993)
I20260619 14:50:24.365492 8042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefcbcd67ed94c059b3197850754b798" candidate_uuid: "a6e46bb4f46e4d309506a4167f43343f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b"
I20260619 14:50:24.365674 8042 raft_consensus.cc:3037] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:24.365597 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aefcbcd67ed94c059b3197850754b798" candidate_uuid: "a6e46bb4f46e4d309506a4167f43343f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cac0c45467f435a83c18d10cc73685a"
I20260619 14:50:24.365855 8329 raft_consensus.cc:3037] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:24.367810 8042 raft_consensus.cc:2454] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e46bb4f46e4d309506a4167f43343f in term 1.
I20260619 14:50:24.367862 8329 raft_consensus.cc:2454] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e46bb4f46e4d309506a4167f43343f in term 1.
I20260619 14:50:24.368180 8123 leader_election.cc:302] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b840e7d2eb14f1e9cc4094eecbe287b, a6e46bb4f46e4d309506a4167f43343f; no voters:
I20260619 14:50:24.368484 8485 raft_consensus.cc:2781] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:24.368906 8485 raft_consensus.cc:686] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 1 LEADER]: Becoming Leader. State: Replica: a6e46bb4f46e4d309506a4167f43343f, State: Running, Role: LEADER
I20260619 14:50:24.369326 8485 consensus_queue.cc:237] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } }
I20260619 14:50:24.369917 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.369458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.2:34035)
0619 14:50:24.369644 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.369804 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 14:50:24.372705 7907 catalog_manager.cc:5495] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e46bb4f46e4d309506a4167f43343f (127.0.241.2). New cstate: current_term: 1 leader_uuid: "a6e46bb4f46e4d309506a4167f43343f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e46bb4f46e4d309506a4167f43343f" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38485 } health_report { overall_health: HEALTHY } } }
I20260619 14:50:24.432077 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.426957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.0.241.2:38485)
0619 14:50:24.427173 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:24.427179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:24.427197 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:50:24.427339 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:24.427342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:24.427477 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:24.427612 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:24.427965 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.428651 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.429915 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.430360 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.430466 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.430779 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:24.431318 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:24.431534 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:24.431683 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 14:50:24.431783 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:50:24.436208 8329 raft_consensus.cc:1261] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a [term 1 FOLLOWER]: Refusing update from remote peer a6e46bb4f46e4d309506a4167f43343f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:24.436436 8042 raft_consensus.cc:1261] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 1 FOLLOWER]: Refusing update from remote peer a6e46bb4f46e4d309506a4167f43343f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:24.437023 8488 consensus_queue.cc:1035] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cac0c45467f435a83c18d10cc73685a" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:24.437418 8485 consensus_queue.cc:1035] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b840e7d2eb14f1e9cc4094eecbe287b" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:24.439849 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.439318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.1:37397)
0619 14:50:24.439576 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.439664 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260619 14:50:24.440239 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.439588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.3:46967)
0619 14:50:24.439828 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.439948 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 14:50:24.446408 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.445856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.3:41087)
0619 14:50:24.446118 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.446228 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260619 14:50:24.447806 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.447385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.1:57609)
0619 14:50:24.447554 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.447646 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 14:50:24.449802 8492 mvcc.cc:205] Tried to move back new op lower bound from 7298583037683240960 to 7298583037420814336. Current Snapshot: MvccSnapshot[applied={T|T < 7298583037683240960}]
I20260619 14:50:24.451220 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.450803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.241.2:60567)
0619 14:50:24.450918 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.451038 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 14:50:24.453088 8493 mvcc.cc:205] Tried to move back new op lower bound from 7298583037683240960 to 7298583037420814336. Current Snapshot: MvccSnapshot[applied={T|T < 7298583037683240960 or (T in {7298583037683240960})}]
I20260619 14:50:24.459131 7907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:24.459388 7907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:24.464845 7907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d: Sending DeleteTablet for 3 replicas of tablet aefcbcd67ed94c059b3197850754b798
I20260619 14:50:24.465739 8022 tablet_service.cc:1514] Processing DeleteTablet for tablet aefcbcd67ed94c059b3197850754b798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:24 UTC) from {username='oryx', principal='oryx/127.0.241.60@KRBTEST.COM'} at 127.0.0.1:48974
I20260619 14:50:24.465996 8168 tablet_service.cc:1514] Processing DeleteTablet for tablet aefcbcd67ed94c059b3197850754b798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:24 UTC) from {username='oryx', principal='oryx/127.0.241.60@KRBTEST.COM'} at 127.0.0.1:56700
I20260619 14:50:24.466516 8309 tablet_service.cc:1514] Processing DeleteTablet for tablet aefcbcd67ed94c059b3197850754b798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:24 UTC) from {username='oryx', principal='oryx/127.0.241.60@KRBTEST.COM'} at 127.0.0.1:45280
I20260619 14:50:24.467337 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 7920
I20260619 14:50:24.478794 8500 tablet_replica.cc:330] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b: stopping tablet replica
I20260619 14:50:24.479010 8500 raft_consensus.cc:2229] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:24.479338 8500 raft_consensus.cc:2258] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:24.482852 8502 tablet_replica.cc:330] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a: stopping tablet replica
I20260619 14:50:24.484699 8500 ts_tablet_manager.cc:1905] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:24.483336 8502 raft_consensus.cc:2229] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:24.485250 8502 raft_consensus.cc:2258] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:24.488148 8502 ts_tablet_manager.cc:1905] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:24.496179 8501 tablet_replica.cc:330] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f: stopping tablet replica
I20260619 14:50:24.496388 8501 raft_consensus.cc:2229] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:24.496593 8502 ts_tablet_manager.cc:1918] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:24.496744 8502 log.cc:1196] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/aefcbcd67ed94c059b3197850754b798
I20260619 14:50:24.497144 8502 ts_tablet_manager.cc:1939] T aefcbcd67ed94c059b3197850754b798 P 8cac0c45467f435a83c18d10cc73685a: Deleting consensus metadata
I20260619 14:50:24.498231 7893 catalog_manager.cc:4841] TS 8cac0c45467f435a83c18d10cc73685a (127.0.241.3:44993): tablet aefcbcd67ed94c059b3197850754b798 (table test-table [id=cf3f11e2cb584fc1854c55ad217c9ec4]) successfully deleted
I20260619 14:50:24.499565 8501 raft_consensus.cc:2258] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:24.500744 8500 ts_tablet_manager.cc:1918] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:24.500883 8500 log.cc:1196] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/aefcbcd67ed94c059b3197850754b798
I20260619 14:50:24.501026 8501 ts_tablet_manager.cc:1905] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:24.501412 8500 ts_tablet_manager.cc:1939] T aefcbcd67ed94c059b3197850754b798 P 6b840e7d2eb14f1e9cc4094eecbe287b: Deleting consensus metadata
I20260619 14:50:24.502538 7894 catalog_manager.cc:4841] TS 6b840e7d2eb14f1e9cc4094eecbe287b (127.0.241.1:38131): tablet aefcbcd67ed94c059b3197850754b798 (table test-table [id=cf3f11e2cb584fc1854c55ad217c9ec4]) successfully deleted
I20260619 14:50:24.504756 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.468844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.0.241.1:38131)
0619 14:50:24.469038 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:24.469044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:24.482526 (+ 13482us) server_negotiation.cc:407] Connection header received
0619 14:50:24.482813 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:24.482820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:24.482904 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:24.482976 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:24.484986 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.486074 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.488220 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.488896 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.488992 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.496973 (+ 7981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:24.496990 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:24.497010 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:24.497057 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:24.501344 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:24.502328 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:24.502338 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:24.502341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:24.502497 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:24.502977 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:24.502982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:24.502986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:24.503517 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:24.503832 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:24.504197 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 14:50:24.504396 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:50:24.505434 8082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:24.508188 8501 ts_tablet_manager.cc:1918] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:24.508339 8501 log.cc:1196] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/aefcbcd67ed94c059b3197850754b798
I20260619 14:50:24.508880 8501 ts_tablet_manager.cc:1939] T aefcbcd67ed94c059b3197850754b798 P a6e46bb4f46e4d309506a4167f43343f: Deleting consensus metadata
I20260619 14:50:24.510120 7894 catalog_manager.cc:4841] TS a6e46bb4f46e4d309506a4167f43343f (127.0.241.2:38485): tablet aefcbcd67ed94c059b3197850754b798 (table test-table [id=cf3f11e2cb584fc1854c55ad217c9ec4]) successfully deleted
I20260619 14:50:24.580330 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.579806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:57022)
0619 14:50:24.580051 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.580152 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260619 14:50:24.581482 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.581064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33289 (local address 127.0.0.1:57036)
0619 14:50:24.581248 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:24.581329 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:33289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 14:50:24.631577 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 7920
I20260619 14:50:24.655879 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 8099
I20260619 14:50:24.671509 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.656650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.0.241.2:38485)
0619 14:50:24.656866 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:24.656870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:24.656887 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:50:24.657081 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:24.657085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:24.657145 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:24.657233 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:24.658831 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.659755 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.661623 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.662035 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.662125 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.665452 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:24.665462 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:24.665476 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:24.665505 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:24.668628 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:24.669397 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:24.669403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:24.669406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:24.669546 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:24.669931 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:24.669934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:24.669936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:24.670257 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:24.670486 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:24.670910 (+ 424us) server_negotiation.cc:299] Negotiation successful
0619 14:50:24.671062 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 14:50:24.672214 8228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:24.789227 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 8099
I20260619 14:50:24.808480 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 8246
I20260619 14:50:24.821597 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:24.809206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.0.241.3:44993)
0619 14:50:24.809363 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:24.809367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:24.809384 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:50:24.809530 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:24.809534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:24.809593 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:24.809721 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:24.810885 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.811701 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.812931 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:24.813314 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:24.813411 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:24.815989 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:24.816004 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:24.816019 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:24.816052 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:24.819094 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:24.819829 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:24.819836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:24.819838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:24.819967 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:24.820333 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:24.820337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:24.820339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:24.820706 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:24.820960 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:24.821175 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 14:50:24.821332 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:50:24.822270 8369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:24.917933 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 8246
I20260619 14:50:24.934791 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 7585
I20260619 14:50:24.935952 7751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:25.052621 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 7585
W20260619 14:50:25.073053 7892 connection.cc:535] client connection to 127.0.241.61:40459 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:50:25.073432 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 7684
I20260619 14:50:25.074290 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:25.073816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:40459 (local address 127.0.0.1:42748)
0619 14:50:25.074015 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:25.074112 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 14:50:25.074409 7939 generic_service.cc:196] Checking for leaks (request via RPC)
W20260619 14:50:25.074501 7892 consensus_peers.cc:487] T 00000000000000000000000000000000 P cd5adced693746f4b6b64150310bc02d -> Peer 04fac3b86b9f46cda4436820eb3e55fb (127.0.241.61:40459): Couldn't send request to peer 04fac3b86b9f46cda4436820eb3e55fb. Status: Network error: Client connection negotiation failed: client connection to 127.0.241.61:40459: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:50:25.190946 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 7684
2026-06-19T14:50:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9764 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:50:25 dist-test-slave-p3ml krb5kdc[8511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:50:25 dist-test-slave-p3ml krb5kdc[8511](info): set up 2 sockets
Jun 19 14:50:25 dist-test-slave-p3ml krb5kdc[8511](info): commencing operation
krb5kdc: starting...
W20260619 14:50:26.292373 964 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:50:26 dist-test-slave-p3ml krb5kdc[8511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.62.keytab.
Entry for principal oryx/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.62.keytab.
I20260619 14:50:26.496562 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35909
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.62.keytab
--principal=oryx/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:35909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:26.816633 8524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:26.817013 8524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:26.817195 8524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:26.824376 8524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:26.824483 8524 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:50:26.824584 8524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:26.824689 8524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:26.824770 8524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:26.824887 8524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:26.834452 8524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:35909
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.62.keytab
--principal=oryx/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:26.835757 8524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:26.837142 8524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:26.849130 8530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:26.849139 8529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:26.849155 8532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:26.850574 8524 server_base.cc:1034] running on GCE node
I20260619 14:50:26.851364 8524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:26.852737 8524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:26.853960 8524 hybrid_clock.cc:648] HybridClock initialized: now 1781880626853905 us; error 35 us; skew 500 ppm
I20260619 14:50:26.854295 8524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:26 dist-test-slave-p3ml krb5kdc[8511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880626, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:26.857471 8524 init.cc:377] Logged in from keytab as oryx/127.0.241.62@KRBTEST.COM (short username oryx)
I20260619 14:50:26.858919 8524 webserver.cc:460] Webserver started at http://127.0.241.62:46185/ using document root <none> and password file <none>
I20260619 14:50:26.859753 8524 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:26.859865 8524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:26.860195 8524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:26.863240 8524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "77fd6faa0a364a4daf83f669403bac8f"
format_stamp: "Formatted at 2026-06-19 14:50:26 on dist-test-slave-p3ml"
server_key: "dc468f49275034bda31134cf7fc24100"
server_key_iv: "6d32ab04e3374282250ef8ebd3c9dd85"
server_key_version: "encryptionkey@0"
I20260619 14:50:26.864215 8524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "77fd6faa0a364a4daf83f669403bac8f"
format_stamp: "Formatted at 2026-06-19 14:50:26 on dist-test-slave-p3ml"
server_key: "dc468f49275034bda31134cf7fc24100"
server_key_iv: "6d32ab04e3374282250ef8ebd3c9dd85"
server_key_version: "encryptionkey@0"
I20260619 14:50:26.869011 8524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:50:26.872280 8539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:26.873593 8524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 14:50:26.873793 8524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "77fd6faa0a364a4daf83f669403bac8f"
format_stamp: "Formatted at 2026-06-19 14:50:26 on dist-test-slave-p3ml"
server_key: "dc468f49275034bda31134cf7fc24100"
server_key_iv: "6d32ab04e3374282250ef8ebd3c9dd85"
server_key_version: "encryptionkey@0"
I20260619 14:50:26.874017 8524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:26.903927 8524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:26.908378 8524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:26.908695 8524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:26.930280 8524 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:35909
I20260619 14:50:26.930286 8600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:35909 every 8 connection(s)
I20260619 14:50:26.931696 8524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:26.934504 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 8524
I20260619 14:50:26.934721 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:26.935556 964 external_mini_cluster.cc:1428] Setting key f66ca5630d7a1e97893b1ee555e86b2a
I20260619 14:50:26.935968 8601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:26.949944 8601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f: Bootstrap starting.
Jun 19 14:50:26 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:26.953006 8601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:26.954017 8601 log.cc:824] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:26.956688 8601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f: No bootstrap required, opened a new log
I20260619 14:50:26.958142 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:26.937569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.0.241.62:35909)
0619 14:50:26.938271 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:26.938283 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:26.938323 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:50:26.940457 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:26.940476 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:26.940876 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:26.941193 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:26.942904 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:26.944276 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:26.945990 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:26.946694 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:26.946852 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:26.951408 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:26.951415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:26.951478 (+ 63us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:26.951531 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:26.954516 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:26.955143 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:26.955149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:26.955155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:26.955310 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:26.955713 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:26.955720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:26.955727 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:26.956180 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:26.956463 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:26.956891 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 14:50:26.957276 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":342,"threads_started":1}
I20260619 14:50:26.964442 8601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fd6faa0a364a4daf83f669403bac8f" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35909 } }
I20260619 14:50:26.964807 8601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:26.964927 8601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77fd6faa0a364a4daf83f669403bac8f, State: Initialized, Role: FOLLOWER
I20260619 14:50:26.965458 8601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fd6faa0a364a4daf83f669403bac8f" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35909 } }
I20260619 14:50:26.965682 8601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:26.965822 8601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:26.965993 8601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:26.968051 8601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fd6faa0a364a4daf83f669403bac8f" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35909 } }
I20260619 14:50:26.968576 8601 leader_election.cc:302] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77fd6faa0a364a4daf83f669403bac8f; no voters:
I20260619 14:50:26.969274 8601 leader_election.cc:288] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:26.969422 8606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:26.970103 8606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [term 1 LEADER]: Becoming Leader. State: Replica: 77fd6faa0a364a4daf83f669403bac8f, State: Running, Role: LEADER
I20260619 14:50:26.970609 8606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fd6faa0a364a4daf83f669403bac8f" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35909 } }
I20260619 14:50:26.971210 8601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:26.973428 8608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77fd6faa0a364a4daf83f669403bac8f. Latest consensus state: current_term: 1 leader_uuid: "77fd6faa0a364a4daf83f669403bac8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fd6faa0a364a4daf83f669403bac8f" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35909 } } }
I20260619 14:50:26.973356 8607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77fd6faa0a364a4daf83f669403bac8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77fd6faa0a364a4daf83f669403bac8f" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35909 } } }
I20260619 14:50:26.973790 8608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:26.973785 8607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:26.977492 8614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:26.982303 8614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:26.990578 8614 catalog_manager.cc:1347] Generated new cluster ID: 0303c5d2f6744b7facfdd922f4ea4121
I20260619 14:50:26.990698 8614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:26.998620 8614 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:27.000308 8614 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:27.006757 8614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f: Generated new TSK 0
I20260619 14:50:27.007490 8614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab.
Entry for principal oryx/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab.
I20260619 14:50:27.064594 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35909
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab
--principal=oryx/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:27.424629 8627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:27.424882 8627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:27.424993 8627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:27.431993 8627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:27.432090 8627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:27.432183 8627 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:50:27.432271 8627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:50:27.441483 8627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab
--principal=oryx/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.241.62:35909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:27.442652 8627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:27.443961 8627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:27.457265 8635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:27.457293 8633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:27.457365 8632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:27.458532 8627 server_base.cc:1034] running on GCE node
I20260619 14:50:27.459121 8627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:27.460192 8627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:27.461401 8627 hybrid_clock.cc:648] HybridClock initialized: now 1781880627461335 us; error 39 us; skew 500 ppm
I20260619 14:50:27.461694 8627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:27 dist-test-slave-p3ml krb5kdc[8511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880627, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:27.464774 8627 init.cc:377] Logged in from keytab as oryx/127.0.241.1@KRBTEST.COM (short username oryx)
I20260619 14:50:27.466218 8627 webserver.cc:460] Webserver started at http://127.0.241.1:36971/ using document root <none> and password file <none>
I20260619 14:50:27.466835 8627 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:27.466924 8627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:27.467181 8627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:27.469976 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "75f9e0f23b3e4e069c06fd2d301c101e"
format_stamp: "Formatted at 2026-06-19 14:50:27 on dist-test-slave-p3ml"
server_key: "c14ca030fc4f8e7dd305a4d66edc6c89"
server_key_iv: "266b3555e00d7d7f1f2201aabe89e354"
server_key_version: "encryptionkey@0"
I20260619 14:50:27.470647 8627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "75f9e0f23b3e4e069c06fd2d301c101e"
format_stamp: "Formatted at 2026-06-19 14:50:27 on dist-test-slave-p3ml"
server_key: "c14ca030fc4f8e7dd305a4d66edc6c89"
server_key_iv: "266b3555e00d7d7f1f2201aabe89e354"
server_key_version: "encryptionkey@0"
I20260619 14:50:27.475508 8627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 14:50:27.479085 8642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:27.480350 8627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 14:50:27.480511 8627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "75f9e0f23b3e4e069c06fd2d301c101e"
format_stamp: "Formatted at 2026-06-19 14:50:27 on dist-test-slave-p3ml"
server_key: "c14ca030fc4f8e7dd305a4d66edc6c89"
server_key_iv: "266b3555e00d7d7f1f2201aabe89e354"
server_key_version: "encryptionkey@0"
I20260619 14:50:27.480729 8627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:27.497313 8627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:27.501036 8627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:27.501300 8627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:27.503772 8627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:27.503873 8627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:27.504050 8627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:27.504128 8627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:50:27 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880627, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:27.526839 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:27.504207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:57263 (local address 127.0.241.62:35909)
0619 14:50:27.504976 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:27.504984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:27.506017 (+ 1033us) server_negotiation.cc:407] Connection header received
0619 14:50:27.508926 (+ 2909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:27.508931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:27.509012 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:27.509151 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:27.511350 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:27.512638 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:27.515012 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:27.515622 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:27.515756 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:27.520284 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:27.520294 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:27.520297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:27.520327 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:27.523610 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:27.524490 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:27.524497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:27.524503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:27.524656 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:27.525244 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:27.525248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:27.525251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:27.525493 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:27.525641 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:27.526283 (+ 642us) server_negotiation.cc:299] Negotiation successful
0619 14:50:27.526489 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":542,"threads_started":1}
I20260619 14:50:27.527850 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:27.504831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35909 (local address 127.0.241.1:57263)
0619 14:50:27.505764 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:27.505808 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:27.508575 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:27.509464 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:27.509477 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:27.510291 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:27.511129 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:27.511151 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:27.512844 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:27.512848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:27.514870 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:27.514881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:27.515867 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:27.515874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:27.516080 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:27.516768 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:27.516804 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:27.520056 (+ 3252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:27.523743 (+ 3687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:27.523751 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:27.523765 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:27.524336 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:27.524798 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:27.524806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:27.524810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:27.525088 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:27.525628 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:27.525635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:27.525938 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 14:50:27.526186 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 14:50:27.526651 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":347,"threads_started":1}
I20260619 14:50:27.547179 8708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:27.533857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35909 (local address 127.0.241.1:59589)
0619 14:50:27.534438 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:27.534460 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:27.534603 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:27.534938 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:27.534942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:27.535351 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:27.535769 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:27.535781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:27.536751 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:27.536756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:27.537772 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:27.537782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:27.538542 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:27.538548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:27.538713 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:27.539323 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:27.539338 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:27.540404 (+ 1066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:27.544413 (+ 4009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:27.544420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:27.544428 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:27.544876 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:27.545365 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:27.545370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:27.545373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:27.545583 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:27.546281 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:27.546292 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:27.546474 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 14:50:27.546587 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 14:50:27.546796 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":345,"threads_started":1}
I20260619 14:50:27.547590 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:27.533764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:59589 (local address 127.0.241.62:35909)
0619 14:50:27.533905 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:27.533910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:27.534540 (+ 630us) server_negotiation.cc:407] Connection header received
0619 14:50:27.534715 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:27.534724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:27.534806 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:27.534900 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:27.535904 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:27.536601 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:27.537902 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:27.538357 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:27.538523 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:27.540622 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:27.540626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:27.540629 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:27.540657 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:27.544270 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:27.545091 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:27.545097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:27.545101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:27.545252 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:27.545724 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:27.545731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:27.545736 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:27.546144 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:27.546303 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:27.546757 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 14:50:27.547069 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:50:27.560858 8627 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:41517
I20260619 14:50:27.560899 8762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:41517 every 8 connection(s)
I20260619 14:50:27.562037 8627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:27.563968 8763 heartbeater.cc:347] Connected to a master server at 127.0.241.62:35909
I20260619 14:50:27.564179 8763 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:27.564774 8763 heartbeater.cc:511] Master 127.0.241.62:35909 requested a full tablet report, sending...
I20260619 14:50:27.566654 8556 ts_manager.cc:194] Registered new tserver with Master: 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517)
I20260619 14:50:27.567793 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 8627
I20260619 14:50:27.568032 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:27.568895 964 external_mini_cluster.cc:1428] Setting key eb668a1ad665a457f92f8efc44f646a3
I20260619 14:50:27.569067 8556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:57263
WARNING: no policy specified for oryx/127.0.241.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab.
Entry for principal oryx/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab.
I20260619 14:50:27.614452 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35909
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab
--principal=oryx/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:27.926817 8767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:27.927124 8767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:27.927299 8767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:27.937243 8767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:27.937373 8767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:27.937502 8767 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:50:27.937633 8767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:50:27.952733 8767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab
--principal=oryx/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.241.62:35909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:27.953967 8767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:27.955176 8767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:27.967698 8772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:27.967669 8773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:27.968016 8775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:27.968703 8767 server_base.cc:1034] running on GCE node
I20260619 14:50:27.969305 8767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:27.970119 8767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:27.971347 8767 hybrid_clock.cc:648] HybridClock initialized: now 1781880627971284 us; error 41 us; skew 500 ppm
I20260619 14:50:27.971731 8767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:27 dist-test-slave-p3ml krb5kdc[8511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880627, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:27.974779 8767 init.cc:377] Logged in from keytab as oryx/127.0.241.2@KRBTEST.COM (short username oryx)
I20260619 14:50:27.976111 8767 webserver.cc:460] Webserver started at http://127.0.241.2:44499/ using document root <none> and password file <none>
I20260619 14:50:27.976824 8767 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:27.976946 8767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:27.977312 8767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:27.980217 8767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "87468e6ac1c6461eb74992f53b8c6a32"
format_stamp: "Formatted at 2026-06-19 14:50:27 on dist-test-slave-p3ml"
server_key: "d101eb2e572c0d57b0eb485ed3e4d66b"
server_key_iv: "c9438a503d4329bae8ba427e6d7f4452"
server_key_version: "encryptionkey@0"
I20260619 14:50:27.980981 8767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "87468e6ac1c6461eb74992f53b8c6a32"
format_stamp: "Formatted at 2026-06-19 14:50:27 on dist-test-slave-p3ml"
server_key: "d101eb2e572c0d57b0eb485ed3e4d66b"
server_key_iv: "c9438a503d4329bae8ba427e6d7f4452"
server_key_version: "encryptionkey@0"
I20260619 14:50:27.985764 8767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 14:50:27.989003 8782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:27.990348 8767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:50:27.990526 8767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "87468e6ac1c6461eb74992f53b8c6a32"
format_stamp: "Formatted at 2026-06-19 14:50:27 on dist-test-slave-p3ml"
server_key: "d101eb2e572c0d57b0eb485ed3e4d66b"
server_key_iv: "c9438a503d4329bae8ba427e6d7f4452"
server_key_version: "encryptionkey@0"
I20260619 14:50:27.990757 8767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:28.006752 8767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:28.010849 8767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:28.011111 8767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:28.013607 8767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:28.013725 8767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:28.013892 8767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:28.013968 8767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:50:28 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880627, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:28.037362 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.014281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:49183 (local address 127.0.241.62:35909)
0619 14:50:28.014462 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.014468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.015870 (+ 1402us) server_negotiation.cc:407] Connection header received
0619 14:50:28.018788 (+ 2918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.018797 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.018887 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.019035 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:28.021128 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.022139 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.023786 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.024333 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.024502 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.029891 (+ 5389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:28.029900 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:28.029907 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:28.029966 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:28.033513 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:28.034585 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:28.034593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:28.034599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:28.034767 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:28.035250 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:28.035255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:28.035258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:28.035661 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:28.035813 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.036785 (+ 972us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.036998 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:50:28.038846 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.014813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35909 (local address 127.0.241.2:49183)
0619 14:50:28.015688 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:28.015747 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:28.018448 (+ 2701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:28.019295 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:28.019309 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:28.020122 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:28.020951 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.020972 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.022295 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.022299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.023637 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.023647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.024620 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.024629 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.025018 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.025818 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:28.025853 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:28.029661 (+ 3808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:28.033680 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:28.033689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:28.033708 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:28.034387 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:28.034907 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:28.034910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:28.034913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:28.035118 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:28.035885 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:28.035891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:28.036395 (+ 504us) client_negotiation.cc:769] Sending connection context
0619 14:50:28.036704 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 14:50:28.037229 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":324,"threads_started":1}
I20260619 14:50:28.060505 8767 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:38643
I20260619 14:50:28.060546 8901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:38643 every 8 connection(s)
I20260619 14:50:28.062016 8767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:28.063395 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 8767
I20260619 14:50:28.063655 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:28.064249 964 external_mini_cluster.cc:1428] Setting key fb2bc1047d06277d9ac16274f9cefc41
I20260619 14:50:28.064999 8902 heartbeater.cc:347] Connected to a master server at 127.0.241.62:35909
I20260619 14:50:28.066434 8902 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:28.067164 8902 heartbeater.cc:511] Master 127.0.241.62:35909 requested a full tablet report, sending...
I20260619 14:50:28.068835 8556 ts_manager.cc:194] Registered new tserver with Master: 87468e6ac1c6461eb74992f53b8c6a32 (127.0.241.2:38643)
I20260619 14:50:28.069417 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.048943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:53311 (local address 127.0.241.62:35909)
0619 14:50:28.049158 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.049167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.052413 (+ 3246us) server_negotiation.cc:407] Connection header received
0619 14:50:28.052596 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.052599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.052661 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.052749 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:28.054368 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.055600 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.057118 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.057863 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.058017 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.060648 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:28.060658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:28.060666 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:28.060712 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:28.064337 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:28.065706 (+ 1369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:28.065710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:28.065712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:28.065910 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:28.067526 (+ 1616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:28.067531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:28.067535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:28.067988 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:28.068137 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.068757 (+ 620us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.069020 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:50:28.069481 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.048974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35909 (local address 127.0.241.2:53311)
0619 14:50:28.052281 (+ 3307us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:28.052306 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:28.052471 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:28.052808 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:28.052816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:28.053646 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:28.054171 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.054191 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.055761 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.055766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.056977 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.056988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.058110 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.058122 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.058306 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.058973 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:28.059008 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:28.060440 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:28.065061 (+ 4621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:28.065069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:28.065073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:28.065594 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:28.066027 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:28.066031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:28.066033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:28.066155 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:28.068199 (+ 2044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:28.068226 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:28.068431 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 14:50:28.068616 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 14:50:28.068967 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3136,"thread_start_us":2955,"threads_started":1}
I20260619 14:50:28.070056 8556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:49183
WARNING: no policy specified for oryx/127.0.241.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab.
Entry for principal oryx/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab.
I20260619 14:50:28.110162 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35909
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab
--principal=oryx/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:28.444159 8906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:28.444475 8906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:28.444651 8906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:28.452121 8906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:28.452281 8906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:28.452432 8906 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:50:28.452579 8906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:50:28.462667 8906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab
--principal=oryx/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.241.62:35909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:28.464071 8906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:28.465413 8906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:28.478547 8911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:28.478667 8914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:28.478693 8912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:28.479540 8906 server_base.cc:1034] running on GCE node
I20260619 14:50:28.480123 8906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:28.480985 8906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:28.482249 8906 hybrid_clock.cc:648] HybridClock initialized: now 1781880628482184 us; error 46 us; skew 500 ppm
I20260619 14:50:28.482548 8906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:28 dist-test-slave-p3ml krb5kdc[8511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880628, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:28.485834 8906 init.cc:377] Logged in from keytab as oryx/127.0.241.3@KRBTEST.COM (short username oryx)
I20260619 14:50:28.487162 8906 webserver.cc:460] Webserver started at http://127.0.241.3:41715/ using document root <none> and password file <none>
I20260619 14:50:28.487903 8906 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:28.488034 8906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:28.488365 8906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:28.491091 8906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "8f8469718c744b99b6001d1a3148037c"
format_stamp: "Formatted at 2026-06-19 14:50:28 on dist-test-slave-p3ml"
server_key: "b255a824081c14f0b55e51dcbad84039"
server_key_iv: "4307c1a92c0a4326bd0bb1bd9654c4f6"
server_key_version: "encryptionkey@0"
I20260619 14:50:28.491897 8906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "8f8469718c744b99b6001d1a3148037c"
format_stamp: "Formatted at 2026-06-19 14:50:28 on dist-test-slave-p3ml"
server_key: "b255a824081c14f0b55e51dcbad84039"
server_key_iv: "4307c1a92c0a4326bd0bb1bd9654c4f6"
server_key_version: "encryptionkey@0"
I20260619 14:50:28.496635 8906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 14:50:28.499862 8921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:28.501037 8906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:50:28.501211 8906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "8f8469718c744b99b6001d1a3148037c"
format_stamp: "Formatted at 2026-06-19 14:50:28 on dist-test-slave-p3ml"
server_key: "b255a824081c14f0b55e51dcbad84039"
server_key_iv: "4307c1a92c0a4326bd0bb1bd9654c4f6"
server_key_version: "encryptionkey@0"
I20260619 14:50:28.501431 8906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:28.525369 8906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:28.529570 8906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:28.529830 8906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:28.532640 8906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:28.532755 8906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:28.532908 8906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:28.533026 8906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 14:50:28 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880628, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:28.557945 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.533042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:58235 (local address 127.0.241.62:35909)
0619 14:50:28.533209 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.533213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.534593 (+ 1380us) server_negotiation.cc:407] Connection header received
0619 14:50:28.536668 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.536674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.536794 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.536904 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:28.539009 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.540374 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.543337 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.543969 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.544109 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.549638 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:28.549646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:28.549653 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:28.549692 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:28.553454 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:28.554598 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:28.554606 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:28.554610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:28.554776 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:28.555482 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:28.555488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:28.555492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:28.555919 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:28.556089 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.557112 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.557405 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:50:28.559116 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.533674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35909 (local address 127.0.241.3:58235)
0619 14:50:28.534432 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:28.534475 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:28.536369 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:28.537122 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:28.537130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:28.537849 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:28.538779 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.538812 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.540626 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.540634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.543143 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.543159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.544198 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.544205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.544535 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.545417 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:28.545462 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:28.549342 (+ 3880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:28.553624 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:28.553641 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:28.553669 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:28.554455 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:28.554926 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:28.554930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:28.554935 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:28.555255 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:28.556113 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:28.556123 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:28.556625 (+ 502us) client_negotiation.cc:769] Sending connection context
0619 14:50:28.557012 (+ 387us) client_negotiation.cc:240] Negotiation successful
0619 14:50:28.557664 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":350,"threads_started":1}
I20260619 14:50:28.573633 8763 heartbeater.cc:503] Master 127.0.241.62:35909 was elected leader, sending a full tablet report...
I20260619 14:50:28.582752 8906 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:32919
I20260619 14:50:28.582774 9040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:32919 every 8 connection(s)
I20260619 14:50:28.584129 8906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:28.586254 9041 heartbeater.cc:347] Connected to a master server at 127.0.241.62:35909
I20260619 14:50:28.586486 9041 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:28.587131 9041 heartbeater.cc:511] Master 127.0.241.62:35909 requested a full tablet report, sending...
I20260619 14:50:28.587409 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.572043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:55787 (local address 127.0.241.62:35909)
0619 14:50:28.572218 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.572224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.572908 (+ 684us) server_negotiation.cc:407] Connection header received
0619 14:50:28.573127 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.573136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.573249 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.573362 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:28.575115 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.576176 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.578541 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.579137 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.579264 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.581522 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:28.581527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:28.581531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:28.581566 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:28.584483 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:28.585374 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:28.585382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:28.585387 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:28.585538 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:28.585954 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:28.585959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:28.585963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:28.586299 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:28.586446 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.586835 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.587050 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:50:28.587584 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.572069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35909 (local address 127.0.241.3:55787)
0619 14:50:28.572775 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:28.572795 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:28.572971 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:28.574034 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:28.574039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:28.574674 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:28.574971 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.574979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.576387 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.576399 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.578395 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.578406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.579316 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.579322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.579532 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.580423 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:28.580439 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:28.581290 (+ 851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:28.584649 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:28.584654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:28.584658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:28.585240 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:28.585678 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:28.585685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:28.585692 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:28.585851 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:28.586413 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:28.586418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:28.586606 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 14:50:28.586750 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 14:50:28.587060 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":398,"threads_started":1}
I20260619 14:50:28.588531 8556 ts_manager.cc:194] Registered new tserver with Master: 8f8469718c744b99b6001d1a3148037c (127.0.241.3:32919)
I20260619 14:50:28.589294 8556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:58235
I20260619 14:50:28.589929 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 8906
I20260619 14:50:28.590122 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:28.590690 964 external_mini_cluster.cc:1428] Setting key 987f820e22363eda9f747bf690f26a13
I20260619 14:50:28.593643 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:50:28 dist-test-slave-p3ml krb5kdc[8511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:50:28 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:28 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:50:28.639963 8651 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:28.630003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.0.241.62:35909)
0619 14:50:28.630239 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.630245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.630435 (+ 190us) server_negotiation.cc:407] Connection header received
0619 14:50:28.630676 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.630680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.630749 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.630844 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:28.632143 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.632890 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.634034 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.634429 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.634521 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.639695 (+ 5174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47490: BlockingRecv error: recv got EOF from 127.0.0.1:47490 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 14:50:28 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:28.655802 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.643150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.0.241.62:35909)
0619 14:50:28.643344 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.643352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.643693 (+ 341us) server_negotiation.cc:407] Connection header received
0619 14:50:28.643810 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.643813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.643879 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.643955 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:28.644994 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.645668 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.646805 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.647168 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.647267 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.650642 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:28.650646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:28.650649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:28.650685 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:28.653275 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:28.654011 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:28.654015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:28.654017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:28.654143 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:28.654484 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:28.654488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:28.654490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:28.654779 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:28.654968 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.655228 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.655405 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:50:28.659502 8556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:28.666349 8556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:28.691335 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.678022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.0.241.2:38643)
0619 14:50:28.683541 (+ 5519us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.683554 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.683575 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:50:28.683641 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.683645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.683830 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.684042 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:28.684831 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.686247 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.688925 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.689805 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.689947 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.690149 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.690842 (+ 693us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.691012 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5365,"thread_start_us":5234,"threads_started":1}
I20260619 14:50:28.691623 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.679398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.0.241.1:41517)
0619 14:50:28.680066 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.680078 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.682807 (+ 2729us) server_negotiation.cc:407] Connection header received
0619 14:50:28.682911 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.682918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.683195 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.683411 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:28.684532 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.686039 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.688386 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.689832 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.689948 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.690133 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.691146 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.691301 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":278,"threads_started":1}
I20260619 14:50:28.691466 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.677826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:38643 (local address 127.0.0.1:60322)
0619 14:50:28.678706 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:28.678761 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:28.678968 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:28.684030 (+ 5062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:28.684035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:28.684059 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:28.684576 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.684588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.686398 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.686406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.688775 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.688785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.690336 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.690342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.690590 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.690653 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 14:50:28.690797 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 14:50:28.691020 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":129,"thread_start_us":405,"threads_started":1}
I20260619 14:50:28.692497 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.678929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:41517 (local address 127.0.0.1:46168)
0619 14:50:28.682519 (+ 3590us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:28.682538 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:28.682656 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:28.683741 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:28.683753 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:28.683820 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:28.684292 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.684316 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.686179 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.686184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.688247 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.688258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.690293 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.690300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.690533 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.690625 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 14:50:28.690800 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 14:50:28.691017 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3458,"mutex_wait_us":65,"thread_start_us":3335,"threads_started":1}
I20260619 14:50:28.693198 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.684135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:32919 (local address 127.0.0.1:50132)
0619 14:50:28.685175 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:28.685191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:28.685384 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:28.686179 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:28.686184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:28.686209 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:28.686696 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.686707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.688671 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.688676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.690721 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.690733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.692467 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.692474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.692652 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.692686 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:50:28.692793 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:50:28.692902 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":803,"threads_started":1}
I20260619 14:50:28.693969 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.682903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.0.241.3:32919)
0619 14:50:28.684756 (+ 1853us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.684767 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.685289 (+ 522us) server_negotiation.cc:407] Connection header received
0619 14:50:28.685542 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.685549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.685744 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.685898 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:28.686901 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.688536 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.690904 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.692257 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.692399 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.692607 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.693474 (+ 867us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.693642 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1674,"thread_start_us":1505,"threads_started":1}
I20260619 14:50:28.695842 8832 tablet_service.cc:1467] Processing CreateTablet for tablet ff1e2ceeee5042d7a27b59533aa32f7b (DEFAULT_TABLE table=test-table [id=21bb72f22dc542169decf86811973f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:28.696074 8693 tablet_service.cc:1467] Processing CreateTablet for tablet ff1e2ceeee5042d7a27b59533aa32f7b (DEFAULT_TABLE table=test-table [id=21bb72f22dc542169decf86811973f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:28.697170 8693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1e2ceeee5042d7a27b59533aa32f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:28.697239 8832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1e2ceeee5042d7a27b59533aa32f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:28.697116 8971 tablet_service.cc:1467] Processing CreateTablet for tablet ff1e2ceeee5042d7a27b59533aa32f7b (DEFAULT_TABLE table=test-table [id=21bb72f22dc542169decf86811973f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:28.698326 8971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff1e2ceeee5042d7a27b59533aa32f7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:28.709244 9065 tablet_bootstrap.cc:492] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e: Bootstrap starting.
I20260619 14:50:28.711102 9066 tablet_bootstrap.cc:492] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32: Bootstrap starting.
I20260619 14:50:28.712586 9067 tablet_bootstrap.cc:492] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c: Bootstrap starting.
I20260619 14:50:28.713423 9065 tablet_bootstrap.cc:654] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:28.714326 9066 tablet_bootstrap.cc:654] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:28.714694 9065 log.cc:824] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:28.715337 9067 tablet_bootstrap.cc:654] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:28.715587 9066 log.cc:824] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:28.716542 9067 log.cc:824] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:28.718286 9066 tablet_bootstrap.cc:492] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32: No bootstrap required, opened a new log
I20260619 14:50:28.718287 9065 tablet_bootstrap.cc:492] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e: No bootstrap required, opened a new log
I20260619 14:50:28.718535 9065 ts_tablet_manager.cc:1397] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 14:50:28.718559 9066 ts_tablet_manager.cc:1397] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260619 14:50:28.720566 9067 tablet_bootstrap.cc:492] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c: No bootstrap required, opened a new log
I20260619 14:50:28.720867 9067 ts_tablet_manager.cc:1397] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 14:50:28.727911 9067 raft_consensus.cc:348] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.728274 9067 raft_consensus.cc:374] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:28.728366 9067 raft_consensus.cc:729] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8469718c744b99b6001d1a3148037c, State: Initialized, Role: FOLLOWER
I20260619 14:50:28.728971 9067 consensus_queue.cc:260] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.729014 9066 raft_consensus.cc:348] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.729427 9066 raft_consensus.cc:374] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:28.729529 9066 raft_consensus.cc:729] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87468e6ac1c6461eb74992f53b8c6a32, State: Initialized, Role: FOLLOWER
I20260619 14:50:28.729422 9065 raft_consensus.cc:348] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.729815 9065 raft_consensus.cc:374] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:28.729929 9065 raft_consensus.cc:729] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f9e0f23b3e4e069c06fd2d301c101e, State: Initialized, Role: FOLLOWER
I20260619 14:50:28.730230 9066 consensus_queue.cc:260] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.730450 9065 consensus_queue.cc:260] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.733103 9065 ts_tablet_manager.cc:1428] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 14:50:28.734134 9041 heartbeater.cc:503] Master 127.0.241.62:35909 was elected leader, sending a full tablet report...
I20260619 14:50:28.734642 9067 ts_tablet_manager.cc:1428] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c: Time spent starting tablet: real 0.014s user 0.003s sys 0.010s
I20260619 14:50:28.735383 8902 heartbeater.cc:503] Master 127.0.241.62:35909 was elected leader, sending a full tablet report...
I20260619 14:50:28.735988 9066 ts_tablet_manager.cc:1428] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260619 14:50:28.815351 8903 tablet.cc:2369] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:50:28.815331 8764 tablet.cc:2369] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:28.833585 9072 raft_consensus.cc:482] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:28.833937 9072 raft_consensus.cc:504] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.835695 9072 leader_election.cc:288] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517), 8f8469718c744b99b6001d1a3148037c (127.0.241.3:32919)
W20260619 14:50:28.835898 9042 tablet.cc:2369] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:28.842991 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.836058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:54049 (local address 127.0.241.1:41517)
0619 14:50:28.836286 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.836292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.836892 (+ 600us) server_negotiation.cc:407] Connection header received
0619 14:50:28.837107 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.837112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.837205 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.837320 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:28.837970 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.838875 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.840971 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.842027 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.842128 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.842235 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.842599 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.842727 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:50:28.843003 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.837067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:33543 (local address 127.0.241.3:32919)
0619 14:50:28.837249 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.837257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.837597 (+ 340us) server_negotiation.cc:407] Connection header received
0619 14:50:28.837713 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.837720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.837798 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.837897 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:28.838470 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.839262 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.841149 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.842107 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.842245 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.842344 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.842658 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.842780 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:50:28.843104 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.836910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:32919 (local address 127.0.241.2:33543)
0619 14:50:28.837503 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:28.837516 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:28.837615 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:28.837943 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:28.837948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:28.837970 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:28.838333 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.838343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.839394 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.839397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.840998 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.841011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.842333 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.842338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.842485 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.842507 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:50:28.842622 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 14:50:28.842736 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":17,"thread_start_us":281,"threads_started":1}
I20260619 14:50:28.843477 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.836049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:41517 (local address 127.0.241.2:54049)
0619 14:50:28.836774 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:28.836794 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:28.836934 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:28.837346 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:28.837351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:28.837378 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:28.837807 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.837821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.839005 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.839010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.840766 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.840810 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.842237 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.842244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.842428 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.842467 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:50:28.842537 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:50:28.842657 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":3,"thread_start_us":404,"threads_started":1}
I20260619 14:50:28.844090 8991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1e2ceeee5042d7a27b59533aa32f7b" candidate_uuid: "87468e6ac1c6461eb74992f53b8c6a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f8469718c744b99b6001d1a3148037c" is_pre_election: true
I20260619 14:50:28.844422 8718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1e2ceeee5042d7a27b59533aa32f7b" candidate_uuid: "87468e6ac1c6461eb74992f53b8c6a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" is_pre_election: true
I20260619 14:50:28.844570 8991 raft_consensus.cc:2454] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87468e6ac1c6461eb74992f53b8c6a32 in term 0.
I20260619 14:50:28.844846 8718 raft_consensus.cc:2454] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87468e6ac1c6461eb74992f53b8c6a32 in term 0.
I20260619 14:50:28.845197 8785 leader_election.cc:302] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87468e6ac1c6461eb74992f53b8c6a32, 8f8469718c744b99b6001d1a3148037c; no voters:
I20260619 14:50:28.845511 9072 raft_consensus.cc:2781] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:28.845690 9072 raft_consensus.cc:482] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:28.845829 9072 raft_consensus.cc:3037] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:28.848196 9072 raft_consensus.cc:504] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.848759 9072 leader_election.cc:288] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [CANDIDATE]: Term 1 election: Requested vote from peers 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517), 8f8469718c744b99b6001d1a3148037c (127.0.241.3:32919)
I20260619 14:50:28.849184 8718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1e2ceeee5042d7a27b59533aa32f7b" candidate_uuid: "87468e6ac1c6461eb74992f53b8c6a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f9e0f23b3e4e069c06fd2d301c101e"
I20260619 14:50:28.849260 8991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff1e2ceeee5042d7a27b59533aa32f7b" candidate_uuid: "87468e6ac1c6461eb74992f53b8c6a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f8469718c744b99b6001d1a3148037c"
I20260619 14:50:28.849481 8991 raft_consensus.cc:3037] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:28.849383 8718 raft_consensus.cc:3037] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:28.851778 8991 raft_consensus.cc:2454] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87468e6ac1c6461eb74992f53b8c6a32 in term 1.
I20260619 14:50:28.852036 8718 raft_consensus.cc:2454] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87468e6ac1c6461eb74992f53b8c6a32 in term 1.
I20260619 14:50:28.852208 8785 leader_election.cc:302] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87468e6ac1c6461eb74992f53b8c6a32, 8f8469718c744b99b6001d1a3148037c; no voters:
I20260619 14:50:28.852504 9072 raft_consensus.cc:2781] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:28.852953 9072 raft_consensus.cc:686] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 LEADER]: Becoming Leader. State: Replica: 87468e6ac1c6461eb74992f53b8c6a32, State: Running, Role: LEADER
I20260619 14:50:28.853417 9072 consensus_queue.cc:237] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.857789 8554 catalog_manager.cc:5495] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87468e6ac1c6461eb74992f53b8c6a32 (127.0.241.2). New cstate: current_term: 1 leader_uuid: "87468e6ac1c6461eb74992f53b8c6a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } health_report { overall_health: UNKNOWN } } }
I20260619 14:50:28.878367 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.872855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.0.241.2:38643)
0619 14:50:28.873049 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.873056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.873158 (+ 102us) server_negotiation.cc:407] Connection header received
0619 14:50:28.873405 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.873409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.873487 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.873640 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:28.873992 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.874709 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.876107 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.876578 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.876736 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.877126 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:28.877745 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:28.877905 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.878004 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.878080 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:50:28.886245 8991 raft_consensus.cc:1261] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Refusing update from remote peer 87468e6ac1c6461eb74992f53b8c6a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:28.886253 8718 raft_consensus.cc:1261] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Refusing update from remote peer 87468e6ac1c6461eb74992f53b8c6a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:28.887143 9076 consensus_queue.cc:1035] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:28.887493 9072 consensus_queue.cc:1035] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:28.897994 9081 mvcc.cc:205] Tried to move back new op lower bound from 7298583055908343808 to 7298583055787614208. Current Snapshot: MvccSnapshot[applied={T|T < 7298583055908343808}]
I20260619 14:50:28.909570 8555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:28.909871 8555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:28.913194 8555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f: Sending DeleteTablet for 3 replicas of tablet ff1e2ceeee5042d7a27b59533aa32f7b
I20260619 14:50:28.914341 8693 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1e2ceeee5042d7a27b59533aa32f7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:28 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:46168
I20260619 14:50:28.914837 8971 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1e2ceeee5042d7a27b59533aa32f7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:28 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:50132
I20260619 14:50:28.915294 9088 tablet_replica.cc:330] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e: stopping tablet replica
I20260619 14:50:28.915248 8832 tablet_service.cc:1514] Processing DeleteTablet for tablet ff1e2ceeee5042d7a27b59533aa32f7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:28 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:60322
I20260619 14:50:28.915808 9088 raft_consensus.cc:2229] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:28.915864 8554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:50:28.916445 9088 raft_consensus.cc:2258] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:50:28.916483 8554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:28.920177 9088 ts_tablet_manager.cc:1905] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:28.924192 8693 tablet_service.cc:1467] Processing CreateTablet for tablet 5d5b75602ba747bab4bbea7592324f76 (DEFAULT_TABLE table=test-table [id=82d7d1d4e196473fb7137775464ec39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:28.924419 8970 tablet_service.cc:1467] Processing CreateTablet for tablet 5d5b75602ba747bab4bbea7592324f76 (DEFAULT_TABLE table=test-table [id=82d7d1d4e196473fb7137775464ec39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:28.924642 8693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d5b75602ba747bab4bbea7592324f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:28.924909 8970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d5b75602ba747bab4bbea7592324f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:28.928396 9088 ts_tablet_manager.cc:1918] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:28.928615 9088 log.cc:1196] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/ff1e2ceeee5042d7a27b59533aa32f7b
I20260619 14:50:28.929284 9088 ts_tablet_manager.cc:1939] T ff1e2ceeee5042d7a27b59533aa32f7b P 75f9e0f23b3e4e069c06fd2d301c101e: Deleting consensus metadata
I20260619 14:50:28.929754 8831 tablet_service.cc:1467] Processing CreateTablet for tablet 5d5b75602ba747bab4bbea7592324f76 (DEFAULT_TABLE table=test-table [id=82d7d1d4e196473fb7137775464ec39e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:28.930279 8831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d5b75602ba747bab4bbea7592324f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:28.931313 8542 catalog_manager.cc:4841] TS 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517): tablet ff1e2ceeee5042d7a27b59533aa32f7b (table test-table [id=21bb72f22dc542169decf86811973f25]) successfully deleted
I20260619 14:50:28.933790 9067 tablet_bootstrap.cc:492] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c: Bootstrap starting.
I20260619 14:50:28.935664 9067 tablet_bootstrap.cc:654] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:28.946241 9089 tablet_replica.cc:330] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c: stopping tablet replica
I20260619 14:50:28.946731 9089 raft_consensus.cc:2229] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:28.947252 9089 raft_consensus.cc:2258] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:28.947391 9067 tablet_bootstrap.cc:492] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c: No bootstrap required, opened a new log
I20260619 14:50:28.947618 9067 ts_tablet_manager.cc:1397] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260619 14:50:28.948541 9067 raft_consensus.cc:348] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.948838 9067 raft_consensus.cc:374] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:28.948982 9067 raft_consensus.cc:729] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8469718c744b99b6001d1a3148037c, State: Initialized, Role: FOLLOWER
I20260619 14:50:28.949246 9065 tablet_bootstrap.cc:492] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e: Bootstrap starting.
I20260619 14:50:28.949354 9067 consensus_queue.cc:260] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.949671 9066 tablet_bootstrap.cc:492] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32: Bootstrap starting.
I20260619 14:50:28.950397 9089 ts_tablet_manager.cc:1905] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:28.950737 9067 ts_tablet_manager.cc:1428] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:50:28.950891 9065 tablet_bootstrap.cc:654] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:28.951278 9066 tablet_bootstrap.cc:654] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:28.952979 9065 tablet_bootstrap.cc:492] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e: No bootstrap required, opened a new log
I20260619 14:50:28.953166 9065 ts_tablet_manager.cc:1397] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260619 14:50:28.953979 9065 raft_consensus.cc:348] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.954248 9065 raft_consensus.cc:374] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:28.954380 9065 raft_consensus.cc:729] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f9e0f23b3e4e069c06fd2d301c101e, State: Initialized, Role: FOLLOWER
I20260619 14:50:28.954710 9065 consensus_queue.cc:260] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.957222 9065 ts_tablet_manager.cc:1428] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260619 14:50:28.957775 9089 ts_tablet_manager.cc:1918] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:28.957929 9089 log.cc:1196] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/ff1e2ceeee5042d7a27b59533aa32f7b
I20260619 14:50:28.958442 9089 ts_tablet_manager.cc:1939] T ff1e2ceeee5042d7a27b59533aa32f7b P 8f8469718c744b99b6001d1a3148037c: Deleting consensus metadata
I20260619 14:50:28.960095 8542 catalog_manager.cc:4841] TS 8f8469718c744b99b6001d1a3148037c (127.0.241.3:32919): tablet ff1e2ceeee5042d7a27b59533aa32f7b (table test-table [id=21bb72f22dc542169decf86811973f25]) successfully deleted
I20260619 14:50:28.972442 9090 tablet_replica.cc:330] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32: stopping tablet replica
I20260619 14:50:28.972971 9066 tablet_bootstrap.cc:492] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32: No bootstrap required, opened a new log
I20260619 14:50:28.972972 9090 raft_consensus.cc:2229] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:28.973265 9066 ts_tablet_manager.cc:1397] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.004s
I20260619 14:50:28.973780 9090 raft_consensus.cc:2258] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:28.974506 9066 raft_consensus.cc:348] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.974826 9066 raft_consensus.cc:374] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:28.974977 9066 raft_consensus.cc:729] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87468e6ac1c6461eb74992f53b8c6a32, State: Initialized, Role: FOLLOWER
I20260619 14:50:28.975313 9066 consensus_queue.cc:260] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.975864 9090 ts_tablet_manager.cc:1905] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:28.978174 9066 ts_tablet_manager.cc:1428] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 14:50:28.982662 9090 ts_tablet_manager.cc:1918] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:28.982842 9090 log.cc:1196] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/ff1e2ceeee5042d7a27b59533aa32f7b
I20260619 14:50:28.983283 9090 ts_tablet_manager.cc:1939] T ff1e2ceeee5042d7a27b59533aa32f7b P 87468e6ac1c6461eb74992f53b8c6a32: Deleting consensus metadata
I20260619 14:50:28.984833 8542 catalog_manager.cc:4841] TS 87468e6ac1c6461eb74992f53b8c6a32 (127.0.241.2:38643): tablet ff1e2ceeee5042d7a27b59533aa32f7b (table test-table [id=21bb72f22dc542169decf86811973f25]) successfully deleted
I20260619 14:50:28.985673 9071 raft_consensus.cc:482] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:28.985917 9071 raft_consensus.cc:504] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.987666 9071 leader_election.cc:288] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517), 87468e6ac1c6461eb74992f53b8c6a32 (127.0.241.2:38643)
I20260619 14:50:28.993661 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.987945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:41517 (local address 127.0.241.3:56789)
0619 14:50:28.988189 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:28.988209 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:28.988326 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:28.988840 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:28.988846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:28.988874 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:28.989094 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.989108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.990378 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.990386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.991779 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.991793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.992916 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.992921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.993135 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.993158 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:50:28.993270 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:50:28.993397 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 14:50:28.993667 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.987995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:56789 (local address 127.0.241.1:41517)
0619 14:50:28.988232 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.988241 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.988262 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:50:28.988479 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.988485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.988584 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.988676 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:28.989518 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.990249 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.991950 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.992773 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.992910 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.993005 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.993294 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.993414 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:50:28.994366 8718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5b75602ba747bab4bbea7592324f76" candidate_uuid: "8f8469718c744b99b6001d1a3148037c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" is_pre_election: true
I20260619 14:50:28.994591 8718 raft_consensus.cc:2454] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f8469718c744b99b6001d1a3148037c in term 0.
I20260619 14:50:28.995577 8924 leader_election.cc:302] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f9e0f23b3e4e069c06fd2d301c101e, 8f8469718c744b99b6001d1a3148037c; no voters:
I20260619 14:50:28.995992 9071 raft_consensus.cc:2781] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:28.996120 9071 raft_consensus.cc:482] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:28.996234 9071 raft_consensus.cc:3037] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:28.996347 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.988774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:42157 (local address 127.0.241.2:38643)
0619 14:50:28.988933 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:28.988940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:28.989012 (+ 72us) server_negotiation.cc:407] Connection header received
0619 14:50:28.989268 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:28.989273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:28.989334 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:28.989398 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:28.989999 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.991127 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.993737 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.995055 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.995171 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.995263 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:28.995901 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 14:50:28.996036 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:50:28.996526 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:28.988364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:38643 (local address 127.0.241.3:42157)
0619 14:50:28.988910 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:28.988936 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:28.989104 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:28.989452 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:28.989456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:28.989480 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:28.989824 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.989834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.991300 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.991305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.993562 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:28.993576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:28.995524 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:28.995530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:28.995728 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:28.995776 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 14:50:28.995861 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:50:28.995984 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":10,"thread_start_us":576,"threads_started":1}
I20260619 14:50:28.997303 8852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5b75602ba747bab4bbea7592324f76" candidate_uuid: "8f8469718c744b99b6001d1a3148037c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87468e6ac1c6461eb74992f53b8c6a32" is_pre_election: true
I20260619 14:50:28.997692 8852 raft_consensus.cc:2454] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f8469718c744b99b6001d1a3148037c in term 0.
I20260619 14:50:28.998984 9071 raft_consensus.cc:504] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:28.999655 9071 leader_election.cc:288] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [CANDIDATE]: Term 1 election: Requested vote from peers 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517), 87468e6ac1c6461eb74992f53b8c6a32 (127.0.241.2:38643)
I20260619 14:50:29.000108 8852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5b75602ba747bab4bbea7592324f76" candidate_uuid: "8f8469718c744b99b6001d1a3148037c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87468e6ac1c6461eb74992f53b8c6a32"
I20260619 14:50:29.000173 8718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d5b75602ba747bab4bbea7592324f76" candidate_uuid: "8f8469718c744b99b6001d1a3148037c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f9e0f23b3e4e069c06fd2d301c101e"
I20260619 14:50:29.000296 8852 raft_consensus.cc:3037] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:29.000378 8718 raft_consensus.cc:3037] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:29.002418 8852 raft_consensus.cc:2454] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f8469718c744b99b6001d1a3148037c in term 1.
I20260619 14:50:29.002760 8924 leader_election.cc:302] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87468e6ac1c6461eb74992f53b8c6a32, 8f8469718c744b99b6001d1a3148037c; no voters:
I20260619 14:50:29.003008 9071 raft_consensus.cc:2781] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:29.003262 8718 raft_consensus.cc:2454] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f8469718c744b99b6001d1a3148037c in term 1.
I20260619 14:50:29.003386 9071 raft_consensus.cc:686] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 1 LEADER]: Becoming Leader. State: Replica: 8f8469718c744b99b6001d1a3148037c, State: Running, Role: LEADER
I20260619 14:50:29.003796 9071 consensus_queue.cc:237] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.006960 8555 catalog_manager.cc:5495] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f8469718c744b99b6001d1a3148037c (127.0.241.3). New cstate: current_term: 1 leader_uuid: "8f8469718c744b99b6001d1a3148037c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } health_report { overall_health: HEALTHY } } }
Jun 19 14:50:29 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880626, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.62@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:29.056524 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.042420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35909 (local address 127.0.0.1:47498)
0619 14:50:29.043168 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:29.043185 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:29.043309 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:29.043672 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:29.043676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:29.044353 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:29.044621 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:29.044630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.046078 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.046082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:29.047068 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:29.047079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.047903 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.047910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:29.048057 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.048728 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:29.048744 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:29.051088 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:29.054056 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:29.054069 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:29.054076 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:29.054532 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:29.054946 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:29.054949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:29.054951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:29.055079 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:29.055718 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:29.055724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:29.055900 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 14:50:29.056034 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 14:50:29.056216 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":441,"threads_started":1}
I20260619 14:50:29.056834 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.042420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.0.241.62:35909)
0619 14:50:29.042666 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:29.042672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:29.043272 (+ 600us) server_negotiation.cc:407] Connection header received
0619 14:50:29.043454 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:29.043461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:29.043541 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:29.043650 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:29.044764 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.045940 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.047212 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.047707 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.047868 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.051278 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:29.051283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:29.051286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:29.051321 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:29.053927 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:29.054667 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:29.054673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:29.054676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:29.054823 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:29.055181 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:29.055186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:29.055190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:29.055589 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:29.055742 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:29.056159 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 14:50:29.056385 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 14:50:29.062017 8555 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:47498:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:50:29.063182 8555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:29.070794 8693 tablet_service.cc:1467] Processing CreateTablet for tablet 85c539625939466aaecce4b3808dea09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=66445a260ca0481a8567cb9c34ade669]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:50:29.071357 8693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c539625939466aaecce4b3808dea09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:29.078624 9065 tablet_bootstrap.cc:492] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e: Bootstrap starting.
I20260619 14:50:29.079984 8832 tablet_service.cc:1467] Processing CreateTablet for tablet 85c539625939466aaecce4b3808dea09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=66445a260ca0481a8567cb9c34ade669]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:50:29.080314 9065 tablet_bootstrap.cc:654] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:29.080552 8832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c539625939466aaecce4b3808dea09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:29.084705 8971 tablet_service.cc:1467] Processing CreateTablet for tablet 85c539625939466aaecce4b3808dea09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=66445a260ca0481a8567cb9c34ade669]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:50:29.085243 8971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85c539625939466aaecce4b3808dea09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:29.090744 9066 tablet_bootstrap.cc:492] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: Bootstrap starting.
I20260619 14:50:29.090960 9065 tablet_bootstrap.cc:492] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e: No bootstrap required, opened a new log
I20260619 14:50:29.091177 9065 ts_tablet_manager.cc:1397] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260619 14:50:29.092168 9067 tablet_bootstrap.cc:492] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c: Bootstrap starting.
I20260619 14:50:29.092046 9065 raft_consensus.cc:348] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.092409 9065 raft_consensus.cc:374] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:29.092499 9066 tablet_bootstrap.cc:654] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:29.092525 9065 raft_consensus.cc:729] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f9e0f23b3e4e069c06fd2d301c101e, State: Initialized, Role: FOLLOWER
I20260619 14:50:29.092887 9065 consensus_queue.cc:260] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.093394 9073 tablet_replica.cc:439] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } }
I20260619 14:50:29.093736 9073 tablet_replica.cc:442] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:50:29.093954 9067 tablet_bootstrap.cc:654] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:29.097290 9065 ts_tablet_manager.cc:1428] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260619 14:50:29.100633 9067 tablet_bootstrap.cc:492] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c: No bootstrap required, opened a new log
I20260619 14:50:29.100816 9067 ts_tablet_manager.cc:1397] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 14:50:29.101048 9066 tablet_bootstrap.cc:492] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: No bootstrap required, opened a new log
I20260619 14:50:29.101218 9066 ts_tablet_manager.cc:1397] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260619 14:50:29.101527 9067 raft_consensus.cc:348] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.101732 9067 raft_consensus.cc:374] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:29.101811 9067 raft_consensus.cc:729] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8469718c744b99b6001d1a3148037c, State: Initialized, Role: FOLLOWER
I20260619 14:50:29.102051 9066 raft_consensus.cc:348] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.102079 9067 consensus_queue.cc:260] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.102298 9066 raft_consensus.cc:374] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:29.102381 9066 raft_consensus.cc:729] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87468e6ac1c6461eb74992f53b8c6a32, State: Initialized, Role: FOLLOWER
I20260619 14:50:29.102455 9071 tablet_replica.cc:439] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } }
I20260619 14:50:29.102716 9071 tablet_replica.cc:442] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:50:29.102645 9066 consensus_queue.cc:260] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.103096 9076 tablet_replica.cc:439] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } }
I20260619 14:50:29.103322 9066 ts_tablet_manager.cc:1428] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 14:50:29.103478 9076 tablet_replica.cc:442] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:50:29.103943 9067 ts_tablet_manager.cc:1428] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 14:50:29.305442 9076 raft_consensus.cc:482] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:29.305704 9076 raft_consensus.cc:504] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.306334 9076 leader_election.cc:288] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517), 8f8469718c744b99b6001d1a3148037c (127.0.241.3:32919)
I20260619 14:50:29.307034 8991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c539625939466aaecce4b3808dea09" candidate_uuid: "87468e6ac1c6461eb74992f53b8c6a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f8469718c744b99b6001d1a3148037c" is_pre_election: true
I20260619 14:50:29.307029 8718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c539625939466aaecce4b3808dea09" candidate_uuid: "87468e6ac1c6461eb74992f53b8c6a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" is_pre_election: true
I20260619 14:50:29.307286 8991 raft_consensus.cc:2454] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87468e6ac1c6461eb74992f53b8c6a32 in term 0.
I20260619 14:50:29.307365 8718 raft_consensus.cc:2454] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87468e6ac1c6461eb74992f53b8c6a32 in term 0.
I20260619 14:50:29.307824 8785 leader_election.cc:302] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87468e6ac1c6461eb74992f53b8c6a32, 8f8469718c744b99b6001d1a3148037c; no voters:
I20260619 14:50:29.308136 9076 raft_consensus.cc:2781] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:29.308259 9076 raft_consensus.cc:482] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:29.308395 9076 raft_consensus.cc:3037] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:29.310577 9076 raft_consensus.cc:504] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.311081 9076 leader_election.cc:288] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [CANDIDATE]: Term 1 election: Requested vote from peers 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517), 8f8469718c744b99b6001d1a3148037c (127.0.241.3:32919)
I20260619 14:50:29.311646 8718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c539625939466aaecce4b3808dea09" candidate_uuid: "87468e6ac1c6461eb74992f53b8c6a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f9e0f23b3e4e069c06fd2d301c101e"
I20260619 14:50:29.311741 8991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85c539625939466aaecce4b3808dea09" candidate_uuid: "87468e6ac1c6461eb74992f53b8c6a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f8469718c744b99b6001d1a3148037c"
I20260619 14:50:29.311901 8718 raft_consensus.cc:3037] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:29.311950 8991 raft_consensus.cc:3037] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:29.314024 8991 raft_consensus.cc:2454] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87468e6ac1c6461eb74992f53b8c6a32 in term 1.
I20260619 14:50:29.314534 8785 leader_election.cc:302] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87468e6ac1c6461eb74992f53b8c6a32, 8f8469718c744b99b6001d1a3148037c; no voters:
I20260619 14:50:29.314903 9076 raft_consensus.cc:2781] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:29.315112 9076 raft_consensus.cc:686] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 LEADER]: Becoming Leader. State: Replica: 87468e6ac1c6461eb74992f53b8c6a32, State: Running, Role: LEADER
I20260619 14:50:29.315615 8718 raft_consensus.cc:2454] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87468e6ac1c6461eb74992f53b8c6a32 in term 1.
I20260619 14:50:29.315454 9076 consensus_queue.cc:237] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.318511 9072 tablet_replica.cc:439] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: TxnStatusTablet state changed. Reason: New leader 87468e6ac1c6461eb74992f53b8c6a32. Latest consensus state: current_term: 1 leader_uuid: "87468e6ac1c6461eb74992f53b8c6a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } }
I20260619 14:50:29.318820 9072 tablet_replica.cc:442] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:50:29.319523 9107 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:50:29.319804 9107 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:50:29.319940 8555 catalog_manager.cc:5495] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87468e6ac1c6461eb74992f53b8c6a32 (127.0.241.2). New cstate: current_term: 1 leader_uuid: "87468e6ac1c6461eb74992f53b8c6a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } health_report { overall_health: UNKNOWN } } }
I20260619 14:50:29.386471 9071 consensus_queue.cc:1035] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:29.392840 9071 consensus_queue.cc:1035] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [LEADER]: Connected to new peer: Peer: permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:29.394933 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.385530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60354 (local address 127.0.241.2:38643)
0619 14:50:29.385721 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:29.385728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:29.385839 (+ 111us) server_negotiation.cc:407] Connection header received
0619 14:50:29.386105 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:29.386111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:29.386257 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:29.386352 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:29.386794 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.388040 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.390466 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.391128 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.391245 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.393551 (+ 2306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:29.393702 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:29.393813 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:29.394504 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 14:50:29.394600 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:50:29.395330 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.385418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:38643 (local address 127.0.0.1:60354)
0619 14:50:29.385749 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:29.385768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:29.385921 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:29.386403 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:29.386408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:29.386422 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:50:29.386615 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:29.386626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.388321 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.388327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:29.389985 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:29.389999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.391780 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.391785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:29.391974 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.392035 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:50:29.393832 (+ 1797us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:50:29.393852 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:50:29.393941 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:50:29.394102 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260619 14:50:29.398736 8991 raft_consensus.cc:1261] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Refusing update from remote peer 87468e6ac1c6461eb74992f53b8c6a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:29.398841 8718 raft_consensus.cc:1261] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Refusing update from remote peer 87468e6ac1c6461eb74992f53b8c6a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:29.399335 9072 consensus_queue.cc:1035] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:29.399655 9076 consensus_queue.cc:1035] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:29.410439 9073 tablet_replica.cc:439] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e: TxnStatusTablet state changed. Reason: New leader 87468e6ac1c6461eb74992f53b8c6a32. Latest consensus state: current_term: 1 leader_uuid: "87468e6ac1c6461eb74992f53b8c6a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } }
I20260619 14:50:29.411218 9073 tablet_replica.cc:442] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:50:29.412065 9079 mvcc.cc:205] Tried to move back new op lower bound from 7298583058010681344 to 7298583057679183872. Current Snapshot: MvccSnapshot[applied={T|T < 7298583058010681344}]
I20260619 14:50:29.415968 9076 tablet_replica.cc:439] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87468e6ac1c6461eb74992f53b8c6a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } }
I20260619 14:50:29.416244 9076 tablet_replica.cc:442] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:50:29.416145 9071 tablet_replica.cc:439] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c: TxnStatusTablet state changed. Reason: New leader 87468e6ac1c6461eb74992f53b8c6a32. Latest consensus state: current_term: 1 leader_uuid: "87468e6ac1c6461eb74992f53b8c6a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } }
I20260619 14:50:29.416432 9071 tablet_replica.cc:442] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:50:29.416453 9076 tablet_replica.cc:439] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87468e6ac1c6461eb74992f53b8c6a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } }
I20260619 14:50:29.416671 9076 tablet_replica.cc:442] T 85c539625939466aaecce4b3808dea09 P 87468e6ac1c6461eb74992f53b8c6a32: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:50:29.422957 9073 tablet_replica.cc:439] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87468e6ac1c6461eb74992f53b8c6a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } }
I20260619 14:50:29.423174 9073 tablet_replica.cc:442] T 85c539625939466aaecce4b3808dea09 P 75f9e0f23b3e4e069c06fd2d301c101e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:50:29.427517 9071 tablet_replica.cc:439] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87468e6ac1c6461eb74992f53b8c6a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } }
I20260619 14:50:29.427781 9071 tablet_replica.cc:442] T 85c539625939466aaecce4b3808dea09 P 8f8469718c744b99b6001d1a3148037c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:50:29.445288 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.431710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.0.241.3:32919)
0619 14:50:29.439876 (+ 8166us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:29.439880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:29.439895 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:50:29.439979 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:29.439982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:29.440054 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:29.440231 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:29.440725 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.441671 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.443101 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.443542 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.443671 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.444147 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:29.444675 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:29.444837 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:29.444895 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 14:50:29.444981 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8076,"thread_start_us":7942,"threads_started":1}
I20260619 14:50:29.461839 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.455673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:56879 (local address 127.0.241.2:38643)
0619 14:50:29.455897 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:29.455903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:29.456419 (+ 516us) server_negotiation.cc:407] Connection header received
0619 14:50:29.456563 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:29.456569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:29.456655 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:29.456784 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:29.457218 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.458407 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.459751 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.460444 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.460575 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.461002 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:29.461183 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:29.461280 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:29.461447 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 14:50:29.461524 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:50:29.461862 9121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.455670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:38643 (local address 127.0.241.3:56879)
0619 14:50:29.456315 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:29.456330 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:29.456439 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:29.456803 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:29.456806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:29.456815 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:50:29.457070 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:29.457076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.458521 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.458524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:29.459611 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:29.459622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.460662 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.460666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:29.460787 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.460839 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:50:29.461325 (+ 486us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:50:29.461333 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:50:29.461392 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 14:50:29.461492 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":359,"threads_started":1}
I20260619 14:50:29.499074 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.492377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:55267 (local address 127.0.241.3:32919)
0619 14:50:29.492535 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:29.492539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:29.493165 (+ 626us) server_negotiation.cc:407] Connection header received
0619 14:50:29.493407 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:29.493415 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:29.493501 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:29.493591 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:29.494151 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.495047 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.497100 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.497784 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.497907 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.498394 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:29.498499 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:29.498597 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:29.498718 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 14:50:29.498792 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:50:29.499358 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.492409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:32919 (local address 127.0.241.2:55267)
0619 14:50:29.493047 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:29.493070 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:29.493210 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:29.493627 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:29.493631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:29.493644 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:50:29.494016 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:29.494031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.495164 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.495171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:29.496953 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:29.496967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.497937 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.497942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:29.498138 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.498205 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:50:29.498599 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:50:29.498624 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:50:29.498700 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:50:29.498848 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":361,"threads_started":1}
I20260619 14:50:29.538975 8555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:29.539223 8555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:29.542548 8555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f: Sending DeleteTablet for 3 replicas of tablet 5d5b75602ba747bab4bbea7592324f76
I20260619 14:50:29.543682 8693 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5b75602ba747bab4bbea7592324f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:29 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:46168
I20260619 14:50:29.544256 8832 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5b75602ba747bab4bbea7592324f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:29 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:60322
I20260619 14:50:29.545138 8971 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d5b75602ba747bab4bbea7592324f76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:29 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:50132
I20260619 14:50:29.555639 9130 tablet_replica.cc:330] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32: stopping tablet replica
I20260619 14:50:29.555876 9130 raft_consensus.cc:2229] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:29.556159 9130 raft_consensus.cc:2258] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:29.557267 9130 ts_tablet_manager.cc:1905] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:29.564713 9130 ts_tablet_manager.cc:1918] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:50:29.564868 9130 log.cc:1196] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/5d5b75602ba747bab4bbea7592324f76
I20260619 14:50:29.565344 9130 ts_tablet_manager.cc:1939] T 5d5b75602ba747bab4bbea7592324f76 P 87468e6ac1c6461eb74992f53b8c6a32: Deleting consensus metadata
I20260619 14:50:29.576659 8542 catalog_manager.cc:4841] TS 87468e6ac1c6461eb74992f53b8c6a32 (127.0.241.2:38643): tablet 5d5b75602ba747bab4bbea7592324f76 (table test-table [id=82d7d1d4e196473fb7137775464ec39e]) successfully deleted
I20260619 14:50:29.598182 9131 tablet_replica.cc:330] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c: stopping tablet replica
I20260619 14:50:29.598582 9131 raft_consensus.cc:2229] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:29.599155 9129 tablet_replica.cc:330] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e: stopping tablet replica
I20260619 14:50:29.599645 9129 raft_consensus.cc:2229] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:29.600178 9131 raft_consensus.cc:2258] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:29.600482 9129 raft_consensus.cc:2258] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:29.601967 9129 ts_tablet_manager.cc:1905] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:29.612151 9131 ts_tablet_manager.cc:1905] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:29.616519 9129 ts_tablet_manager.cc:1918] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:50:29.623219 9129 log.cc:1196] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/5d5b75602ba747bab4bbea7592324f76
I20260619 14:50:29.623786 9129 ts_tablet_manager.cc:1939] T 5d5b75602ba747bab4bbea7592324f76 P 75f9e0f23b3e4e069c06fd2d301c101e: Deleting consensus metadata
I20260619 14:50:29.624974 8542 catalog_manager.cc:4841] TS 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517): tablet 5d5b75602ba747bab4bbea7592324f76 (table test-table [id=82d7d1d4e196473fb7137775464ec39e]) successfully deleted
I20260619 14:50:29.619396 9131 ts_tablet_manager.cc:1918] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:50:29.635592 9131 log.cc:1196] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/5d5b75602ba747bab4bbea7592324f76
I20260619 14:50:29.636137 9131 ts_tablet_manager.cc:1939] T 5d5b75602ba747bab4bbea7592324f76 P 8f8469718c744b99b6001d1a3148037c: Deleting consensus metadata
I20260619 14:50:29.637351 8542 catalog_manager.cc:4841] TS 8f8469718c744b99b6001d1a3148037c (127.0.241.3:32919): tablet 5d5b75602ba747bab4bbea7592324f76 (table test-table [id=82d7d1d4e196473fb7137775464ec39e]) successfully deleted
Jun 19 14:50:29 dist-test-slave-p3ml krb5kdc[8511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260619 14:50:29.658986 964 mini_kdc.cc:317] Time spent kinit for test-admin: real 0.114s user 0.005s sys 0.010s
Jun 19 14:50:29 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:29 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:50:29.700284 9139 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:29.669582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.0.241.62:35909)
0619 14:50:29.683841 (+ 14259us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:29.683847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:29.683868 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:50:29.683934 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:29.683940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:29.684023 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:29.684143 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:29.686105 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.687295 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.689174 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.689811 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.689947 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.699963 (+ 10016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47506: BlockingRecv error: recv got EOF from 127.0.0.1:47506 (error 108)
Metrics: {"server-negotiator.queue_time_us":14134,"thread_start_us":14004,"threads_started":1}
Jun 19 14:50:29 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM
I20260619 14:50:29.719728 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.702496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.0.241.62:35909)
0619 14:50:29.702661 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:29.702667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:29.703109 (+ 442us) server_negotiation.cc:407] Connection header received
0619 14:50:29.703277 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:29.703283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:29.703360 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:29.703478 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:29.705011 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.706066 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.707953 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.708611 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.708742 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.713417 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:29.713424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:29.713437 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:29.713476 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:29.716871 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:29.717617 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:29.717623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:29.717627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:29.717770 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:29.718178 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:29.718183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:29.718186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:29.718550 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:29.718793 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:29.719107 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 14:50:29.719315 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:50:29.723582 8555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:29.724284 8555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:29.731722 8693 tablet_service.cc:1467] Processing CreateTablet for tablet 9357ad95f015452686eab3976dfd9920 (DEFAULT_TABLE table=test-table [id=22414c29c7d144628432ccb1b9a2b670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:29.732142 8693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9357ad95f015452686eab3976dfd9920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:29.735563 8971 tablet_service.cc:1467] Processing CreateTablet for tablet 9357ad95f015452686eab3976dfd9920 (DEFAULT_TABLE table=test-table [id=22414c29c7d144628432ccb1b9a2b670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:29.735833 8832 tablet_service.cc:1467] Processing CreateTablet for tablet 9357ad95f015452686eab3976dfd9920 (DEFAULT_TABLE table=test-table [id=22414c29c7d144628432ccb1b9a2b670]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:29.735981 8971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9357ad95f015452686eab3976dfd9920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:29.736301 8832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9357ad95f015452686eab3976dfd9920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:29.757589 9147 tablet_bootstrap.cc:492] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e: Bootstrap starting.
I20260619 14:50:29.760298 9146 tablet_bootstrap.cc:492] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c: Bootstrap starting.
I20260619 14:50:29.759414 9147 tablet_bootstrap.cc:654] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:29.763827 9146 tablet_bootstrap.cc:654] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:29.766068 9147 tablet_bootstrap.cc:492] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e: No bootstrap required, opened a new log
I20260619 14:50:29.766204 9147 ts_tablet_manager.cc:1397] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20260619 14:50:29.766947 9147 raft_consensus.cc:348] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.767220 9147 raft_consensus.cc:374] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:29.767309 9147 raft_consensus.cc:729] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f9e0f23b3e4e069c06fd2d301c101e, State: Initialized, Role: FOLLOWER
I20260619 14:50:29.767602 9147 consensus_queue.cc:260] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.768330 9145 tablet_bootstrap.cc:492] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32: Bootstrap starting.
I20260619 14:50:29.768981 9147 ts_tablet_manager.cc:1428] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 14:50:29.770569 9146 tablet_bootstrap.cc:492] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c: No bootstrap required, opened a new log
I20260619 14:50:29.770742 9146 ts_tablet_manager.cc:1397] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260619 14:50:29.771204 9145 tablet_bootstrap.cc:654] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:29.771755 9146 raft_consensus.cc:348] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.772017 9146 raft_consensus.cc:374] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:29.772117 9146 raft_consensus.cc:729] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8469718c744b99b6001d1a3148037c, State: Initialized, Role: FOLLOWER
I20260619 14:50:29.772410 9146 consensus_queue.cc:260] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.773082 9146 ts_tablet_manager.cc:1428] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 14:50:29.779359 9145 tablet_bootstrap.cc:492] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32: No bootstrap required, opened a new log
I20260619 14:50:29.779560 9145 ts_tablet_manager.cc:1397] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260619 14:50:29.780400 9145 raft_consensus.cc:348] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.780670 9145 raft_consensus.cc:374] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:29.780778 9145 raft_consensus.cc:729] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87468e6ac1c6461eb74992f53b8c6a32, State: Initialized, Role: FOLLOWER
I20260619 14:50:29.781055 9145 consensus_queue.cc:260] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.781666 9145 ts_tablet_manager.cc:1428] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 14:50:29.786370 9123 raft_consensus.cc:482] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:29.786569 9123 raft_consensus.cc:504] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.787262 9123 leader_election.cc:288] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517), 87468e6ac1c6461eb74992f53b8c6a32 (127.0.241.2:38643)
I20260619 14:50:29.787799 8852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9357ad95f015452686eab3976dfd9920" candidate_uuid: "8f8469718c744b99b6001d1a3148037c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87468e6ac1c6461eb74992f53b8c6a32" is_pre_election: true
I20260619 14:50:29.787810 8718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9357ad95f015452686eab3976dfd9920" candidate_uuid: "8f8469718c744b99b6001d1a3148037c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" is_pre_election: true
I20260619 14:50:29.788043 8718 raft_consensus.cc:2454] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f8469718c744b99b6001d1a3148037c in term 0.
I20260619 14:50:29.788043 8852 raft_consensus.cc:2454] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f8469718c744b99b6001d1a3148037c in term 0.
I20260619 14:50:29.788486 8924 leader_election.cc:302] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f9e0f23b3e4e069c06fd2d301c101e, 8f8469718c744b99b6001d1a3148037c; no voters:
I20260619 14:50:29.788894 9123 raft_consensus.cc:2781] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:29.789011 9123 raft_consensus.cc:482] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:29.789108 9123 raft_consensus.cc:3037] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:29.792254 9123 raft_consensus.cc:504] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.793200 8718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9357ad95f015452686eab3976dfd9920" candidate_uuid: "8f8469718c744b99b6001d1a3148037c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f9e0f23b3e4e069c06fd2d301c101e"
I20260619 14:50:29.793440 8718 raft_consensus.cc:3037] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:29.793478 8852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9357ad95f015452686eab3976dfd9920" candidate_uuid: "8f8469718c744b99b6001d1a3148037c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87468e6ac1c6461eb74992f53b8c6a32"
I20260619 14:50:29.793685 8852 raft_consensus.cc:3037] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:29.796399 8718 raft_consensus.cc:2454] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f8469718c744b99b6001d1a3148037c in term 1.
I20260619 14:50:29.796738 8852 raft_consensus.cc:2454] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f8469718c744b99b6001d1a3148037c in term 1.
I20260619 14:50:29.797338 8924 leader_election.cc:302] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87468e6ac1c6461eb74992f53b8c6a32, 8f8469718c744b99b6001d1a3148037c; no voters:
I20260619 14:50:29.797613 9124 raft_consensus.cc:2781] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:29.797748 9123 leader_election.cc:288] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [CANDIDATE]: Term 1 election: Requested vote from peers 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517), 87468e6ac1c6461eb74992f53b8c6a32 (127.0.241.2:38643)
I20260619 14:50:29.797786 9124 raft_consensus.cc:686] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 1 LEADER]: Becoming Leader. State: Replica: 8f8469718c744b99b6001d1a3148037c, State: Running, Role: LEADER
I20260619 14:50:29.798115 9124 consensus_queue.cc:237] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } }
I20260619 14:50:29.805279 8555 catalog_manager.cc:5495] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f8469718c744b99b6001d1a3148037c (127.0.241.3). New cstate: current_term: 1 leader_uuid: "8f8469718c744b99b6001d1a3148037c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } health_report { overall_health: HEALTHY } } }
I20260619 14:50:29.820482 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.815201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50156 (local address 127.0.241.3:32919)
0619 14:50:29.815397 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:29.815402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:29.815604 (+ 202us) server_negotiation.cc:407] Connection header received
0619 14:50:29.815806 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:29.815813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:29.815915 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:29.816023 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:29.816383 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.817171 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.818650 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.819131 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.819266 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.819734 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:29.819824 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:29.819922 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:29.820065 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 14:50:29.820150 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:50:29.823170 8718 raft_consensus.cc:1261] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Refusing update from remote peer 8f8469718c744b99b6001d1a3148037c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:29.823138 8852 raft_consensus.cc:1261] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Refusing update from remote peer 8f8469718c744b99b6001d1a3148037c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:29.823839 9071 consensus_queue.cc:1035] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:29.824069 9124 consensus_queue.cc:1035] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [LEADER]: Connected to new peer: Peer: permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:29.844249 9080 mvcc.cc:205] Tried to move back new op lower bound from 7298583059749466112 to 7298583059655983104. Current Snapshot: MvccSnapshot[applied={T|T < 7298583059749466112}]
I20260619 14:50:29.848172 8556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:29.848453 8556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:29.851882 8556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f: Sending DeleteTablet for 3 replicas of tablet 9357ad95f015452686eab3976dfd9920
I20260619 14:50:29.853013 8971 tablet_service.cc:1514] Processing DeleteTablet for tablet 9357ad95f015452686eab3976dfd9920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:29 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:50132
I20260619 14:50:29.853190 8832 tablet_service.cc:1514] Processing DeleteTablet for tablet 9357ad95f015452686eab3976dfd9920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:29 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:60322
I20260619 14:50:29.853322 8693 tablet_service.cc:1514] Processing DeleteTablet for tablet 9357ad95f015452686eab3976dfd9920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:29 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:46168
I20260619 14:50:29.853477 9130 tablet_replica.cc:330] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32: stopping tablet replica
I20260619 14:50:29.853595 9131 tablet_replica.cc:330] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c: stopping tablet replica
I20260619 14:50:29.853674 9130 raft_consensus.cc:2229] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:29.853788 9131 raft_consensus.cc:2229] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:29.853888 9130 raft_consensus.cc:2258] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:29.854103 9131 raft_consensus.cc:2258] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:29.854369 9129 tablet_replica.cc:330] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e: stopping tablet replica
I20260619 14:50:29.854559 9129 raft_consensus.cc:2229] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:29.854746 9129 raft_consensus.cc:2258] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:29.855775 8971 tablet_service.cc:1514] Processing DeleteTablet for tablet 9357ad95f015452686eab3976dfd9920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:29 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:50132
I20260619 14:50:29.856086 9131 ts_tablet_manager.cc:1905] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:29.857179 8556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:29.857774 8556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:29.861565 9130 ts_tablet_manager.cc:1905] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:29.868299 8693 tablet_service.cc:1467] Processing CreateTablet for tablet 7f09b14ad3924451ab5a6290b018524a (DEFAULT_TABLE table=test-table [id=76c30615cde341a7854fc972b60df953]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:29.868450 8832 tablet_service.cc:1467] Processing CreateTablet for tablet 7f09b14ad3924451ab5a6290b018524a (DEFAULT_TABLE table=test-table [id=76c30615cde341a7854fc972b60df953]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:29.868745 8693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f09b14ad3924451ab5a6290b018524a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:29.868791 8832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f09b14ad3924451ab5a6290b018524a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:29.872359 8971 tablet_service.cc:1467] Processing CreateTablet for tablet 7f09b14ad3924451ab5a6290b018524a (DEFAULT_TABLE table=test-table [id=76c30615cde341a7854fc972b60df953]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:29.872767 8971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f09b14ad3924451ab5a6290b018524a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:29.875093 9147 tablet_bootstrap.cc:492] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e: Bootstrap starting.
I20260619 14:50:29.875748 9131 ts_tablet_manager.cc:1918] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:29.875888 9131 log.cc:1196] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/9357ad95f015452686eab3976dfd9920
I20260619 14:50:29.876683 9147 tablet_bootstrap.cc:654] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:29.876758 9131 ts_tablet_manager.cc:1939] T 9357ad95f015452686eab3976dfd9920 P 8f8469718c744b99b6001d1a3148037c: Deleting consensus metadata
I20260619 14:50:29.877547 9129 ts_tablet_manager.cc:1905] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:29.881446 9146 tablet_bootstrap.cc:492] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c: Bootstrap starting.
I20260619 14:50:29.882984 9146 tablet_bootstrap.cc:654] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:29.884552 9129 ts_tablet_manager.cc:1918] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:29.884703 9129 log.cc:1196] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/9357ad95f015452686eab3976dfd9920
I20260619 14:50:29.885488 9129 ts_tablet_manager.cc:1939] T 9357ad95f015452686eab3976dfd9920 P 75f9e0f23b3e4e069c06fd2d301c101e: Deleting consensus metadata
I20260619 14:50:29.885982 9145 tablet_bootstrap.cc:492] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32: Bootstrap starting.
I20260619 14:50:29.885496 9147 tablet_bootstrap.cc:492] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e: No bootstrap required, opened a new log
I20260619 14:50:29.886433 8542 catalog_manager.cc:4841] TS 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517): tablet 9357ad95f015452686eab3976dfd9920 (table test-table [id=22414c29c7d144628432ccb1b9a2b670]) successfully deleted
I20260619 14:50:29.886554 9147 ts_tablet_manager.cc:1397] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260619 14:50:29.887270 9147 raft_consensus.cc:348] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } }
I20260619 14:50:29.887593 9145 tablet_bootstrap.cc:654] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:29.887585 9147 raft_consensus.cc:374] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:29.887665 9147 raft_consensus.cc:729] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f9e0f23b3e4e069c06fd2d301c101e, State: Initialized, Role: FOLLOWER
I20260619 14:50:29.888231 9147 consensus_queue.cc:260] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } }
I20260619 14:50:29.888680 9130 ts_tablet_manager.cc:1918] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:29.888863 9130 log.cc:1196] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/9357ad95f015452686eab3976dfd9920
I20260619 14:50:29.889386 9130 ts_tablet_manager.cc:1939] T 9357ad95f015452686eab3976dfd9920 P 87468e6ac1c6461eb74992f53b8c6a32: Deleting consensus metadata
I20260619 14:50:29.890831 8542 catalog_manager.cc:4841] TS 87468e6ac1c6461eb74992f53b8c6a32 (127.0.241.2:38643): tablet 9357ad95f015452686eab3976dfd9920 (table test-table [id=22414c29c7d144628432ccb1b9a2b670]) successfully deleted
I20260619 14:50:29.891077 9147 ts_tablet_manager.cc:1428] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:50:29.893263 8542 catalog_manager.cc:4841] TS 8f8469718c744b99b6001d1a3148037c (127.0.241.3:32919): tablet 9357ad95f015452686eab3976dfd9920 (table test-table [id=22414c29c7d144628432ccb1b9a2b670]) successfully deleted
W20260619 14:50:29.894248 8542 catalog_manager.cc:4808] TS 8f8469718c744b99b6001d1a3148037c (127.0.241.3:32919): delete failed for tablet 9357ad95f015452686eab3976dfd9920 because the tablet was not found. No further retry: Not found: Tablet not found: 9357ad95f015452686eab3976dfd9920
I20260619 14:50:29.895195 9146 tablet_bootstrap.cc:492] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c: No bootstrap required, opened a new log
I20260619 14:50:29.895377 9146 ts_tablet_manager.cc:1397] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260619 14:50:29.896242 9146 raft_consensus.cc:348] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } }
I20260619 14:50:29.896474 9146 raft_consensus.cc:374] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:29.896559 9146 raft_consensus.cc:729] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8469718c744b99b6001d1a3148037c, State: Initialized, Role: FOLLOWER
I20260619 14:50:29.896818 9146 consensus_queue.cc:260] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } }
I20260619 14:50:29.897387 9146 ts_tablet_manager.cc:1428] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 14:50:29.905035 9145 tablet_bootstrap.cc:492] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32: No bootstrap required, opened a new log
I20260619 14:50:29.905373 9145 ts_tablet_manager.cc:1397] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.000s
I20260619 14:50:29.906438 9145 raft_consensus.cc:348] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } }
I20260619 14:50:29.906790 9145 raft_consensus.cc:374] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:29.906935 9145 raft_consensus.cc:729] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87468e6ac1c6461eb74992f53b8c6a32, State: Initialized, Role: FOLLOWER
I20260619 14:50:29.907202 9145 consensus_queue.cc:260] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } }
I20260619 14:50:29.907783 9145 ts_tablet_manager.cc:1428] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 14:50:29.921053 9072 raft_consensus.cc:482] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:29.921226 9072 raft_consensus.cc:504] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } }
I20260619 14:50:29.921788 9072 leader_election.cc:288] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f8469718c744b99b6001d1a3148037c (127.0.241.3:32919), 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517)
I20260619 14:50:29.922325 8990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f09b14ad3924451ab5a6290b018524a" candidate_uuid: "87468e6ac1c6461eb74992f53b8c6a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f8469718c744b99b6001d1a3148037c" is_pre_election: true
I20260619 14:50:29.922518 8718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f09b14ad3924451ab5a6290b018524a" candidate_uuid: "87468e6ac1c6461eb74992f53b8c6a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" is_pre_election: true
I20260619 14:50:29.922631 8990 raft_consensus.cc:2454] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87468e6ac1c6461eb74992f53b8c6a32 in term 0.
I20260619 14:50:29.922751 8718 raft_consensus.cc:2454] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87468e6ac1c6461eb74992f53b8c6a32 in term 0.
I20260619 14:50:29.923194 8785 leader_election.cc:302] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75f9e0f23b3e4e069c06fd2d301c101e, 87468e6ac1c6461eb74992f53b8c6a32; no voters:
I20260619 14:50:29.923521 9072 raft_consensus.cc:2781] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:29.923652 9072 raft_consensus.cc:482] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:29.923769 9072 raft_consensus.cc:3037] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:29.926205 9072 raft_consensus.cc:504] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } }
I20260619 14:50:29.926749 9072 leader_election.cc:288] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [CANDIDATE]: Term 1 election: Requested vote from peers 8f8469718c744b99b6001d1a3148037c (127.0.241.3:32919), 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517)
I20260619 14:50:29.927217 8990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f09b14ad3924451ab5a6290b018524a" candidate_uuid: "87468e6ac1c6461eb74992f53b8c6a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f8469718c744b99b6001d1a3148037c"
I20260619 14:50:29.927421 8990 raft_consensus.cc:3037] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:29.927479 8718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f09b14ad3924451ab5a6290b018524a" candidate_uuid: "87468e6ac1c6461eb74992f53b8c6a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f9e0f23b3e4e069c06fd2d301c101e"
I20260619 14:50:29.927673 8718 raft_consensus.cc:3037] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:29.930550 8718 raft_consensus.cc:2454] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87468e6ac1c6461eb74992f53b8c6a32 in term 1.
I20260619 14:50:29.930646 8990 raft_consensus.cc:2454] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87468e6ac1c6461eb74992f53b8c6a32 in term 1.
I20260619 14:50:29.931021 8785 leader_election.cc:302] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87468e6ac1c6461eb74992f53b8c6a32, 8f8469718c744b99b6001d1a3148037c; no voters:
I20260619 14:50:29.931298 9072 raft_consensus.cc:2781] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:29.931505 9072 raft_consensus.cc:686] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 LEADER]: Becoming Leader. State: Replica: 87468e6ac1c6461eb74992f53b8c6a32, State: Running, Role: LEADER
I20260619 14:50:29.931800 9072 consensus_queue.cc:237] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } }
I20260619 14:50:29.934296 8556 catalog_manager.cc:5495] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87468e6ac1c6461eb74992f53b8c6a32 (127.0.241.2). New cstate: current_term: 1 leader_uuid: "87468e6ac1c6461eb74992f53b8c6a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87468e6ac1c6461eb74992f53b8c6a32" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38643 } health_report { overall_health: HEALTHY } } }
I20260619 14:50:29.966250 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.961150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.0.241.2:38643)
0619 14:50:29.961655 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:29.961659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:29.961671 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:50:29.961715 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:29.961718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:29.961773 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:29.961875 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:29.962318 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.963069 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.964471 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.964884 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.965029 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.965468 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:29.965569 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:29.965665 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:29.965808 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 14:50:29.965880 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411}
I20260619 14:50:29.978210 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.973574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:38643 (local address 127.0.241.2:40967)
0619 14:50:29.973748 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:29.973766 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:29.973868 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:29.974165 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:29.974169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:29.974179 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:50:29.974349 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:29.974357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.975286 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.975293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:29.976340 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:29.976351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.977165 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.977170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:29.977335 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.977364 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:50:29.977720 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:50:29.977730 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:50:29.977804 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:50:29.977926 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 14:50:29.978210 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:29.973654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:40967 (local address 127.0.241.2:38643)
0619 14:50:29.973790 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:29.973798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:29.973815 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:50:29.973997 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:29.974002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:29.974062 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:29.974144 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:29.974485 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.975150 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.976475 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:29.977008 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:29.977158 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:29.977502 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:29.977608 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:29.977696 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:29.977829 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 14:50:29.977887 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":34}
I20260619 14:50:29.986500 8990 raft_consensus.cc:1261] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Refusing update from remote peer 87468e6ac1c6461eb74992f53b8c6a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:29.986560 8718 raft_consensus.cc:1261] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Refusing update from remote peer 87468e6ac1c6461eb74992f53b8c6a32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:29.987066 9076 consensus_queue.cc:1035] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f9e0f23b3e4e069c06fd2d301c101e" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:29.987320 9072 consensus_queue.cc:1035] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f8469718c744b99b6001d1a3148037c" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:30.043884 8555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:30.044193 8555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:30.047359 8555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77fd6faa0a364a4daf83f669403bac8f: Sending DeleteTablet for 3 replicas of tablet 7f09b14ad3924451ab5a6290b018524a
I20260619 14:50:30.048252 8693 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f09b14ad3924451ab5a6290b018524a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:30 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:46168
I20260619 14:50:30.048236 8971 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f09b14ad3924451ab5a6290b018524a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:30 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:50132
I20260619 14:50:30.048434 8831 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f09b14ad3924451ab5a6290b018524a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:30 UTC) from {username='oryx', principal='oryx/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:60322
I20260619 14:50:30.048545 9129 tablet_replica.cc:330] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e: stopping tablet replica
I20260619 14:50:30.048691 9131 tablet_replica.cc:330] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c: stopping tablet replica
I20260619 14:50:30.048723 9130 tablet_replica.cc:330] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32: stopping tablet replica
I20260619 14:50:30.048730 9129 raft_consensus.cc:2229] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:30.048859 9131 raft_consensus.cc:2229] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:30.048897 9130 raft_consensus.cc:2229] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:30.048944 9129 raft_consensus.cc:2258] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:30.049059 9131 raft_consensus.cc:2258] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:30.049237 9130 raft_consensus.cc:2258] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:30.050110 9131 ts_tablet_manager.cc:1905] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:30.051051 9130 ts_tablet_manager.cc:1905] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:30.051086 9129 ts_tablet_manager.cc:1905] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:30.057412 9131 ts_tablet_manager.cc:1918] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:50:30.057502 9129 ts_tablet_manager.cc:1918] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:50:30.057565 9131 log.cc:1196] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/7f09b14ad3924451ab5a6290b018524a
I20260619 14:50:30.057612 9129 log.cc:1196] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/7f09b14ad3924451ab5a6290b018524a
I20260619 14:50:30.058022 9131 ts_tablet_manager.cc:1939] T 7f09b14ad3924451ab5a6290b018524a P 8f8469718c744b99b6001d1a3148037c: Deleting consensus metadata
I20260619 14:50:30.058033 9129 ts_tablet_manager.cc:1939] T 7f09b14ad3924451ab5a6290b018524a P 75f9e0f23b3e4e069c06fd2d301c101e: Deleting consensus metadata
I20260619 14:50:30.058405 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 8627
I20260619 14:50:30.059338 8542 catalog_manager.cc:4841] TS 75f9e0f23b3e4e069c06fd2d301c101e (127.0.241.1:41517): tablet 7f09b14ad3924451ab5a6290b018524a (table test-table [id=76c30615cde341a7854fc972b60df953]) successfully deleted
I20260619 14:50:30.059685 8542 catalog_manager.cc:4841] TS 8f8469718c744b99b6001d1a3148037c (127.0.241.3:32919): tablet 7f09b14ad3924451ab5a6290b018524a (table test-table [id=76c30615cde341a7854fc972b60df953]) successfully deleted
I20260619 14:50:30.063215 9130 ts_tablet_manager.cc:1918] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:50:30.063424 9130 log.cc:1196] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/7f09b14ad3924451ab5a6290b018524a
I20260619 14:50:30.063969 9130 ts_tablet_manager.cc:1939] T 7f09b14ad3924451ab5a6290b018524a P 87468e6ac1c6461eb74992f53b8c6a32: Deleting consensus metadata
I20260619 14:50:30.065397 8542 catalog_manager.cc:4841] TS 87468e6ac1c6461eb74992f53b8c6a32 (127.0.241.2:38643): tablet 7f09b14ad3924451ab5a6290b018524a (table test-table [id=76c30615cde341a7854fc972b60df953]) successfully deleted
Jun 19 14:50:30 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.241.1@KRBTEST.COM
I20260619 14:50:30.081540 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:30.059280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.0.241.1:41517)
0619 14:50:30.066801 (+ 7521us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:30.066805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:30.066819 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:50:30.066862 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:30.066865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:30.066921 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:30.067055 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:30.068596 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:30.069317 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:30.070544 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:30.070969 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:30.071106 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:30.075169 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:30.075183 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:30.075198 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:30.075228 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:30.078704 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:30.079396 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:30.079406 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:30.079410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:30.079633 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:30.080004 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:30.080010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:30.080013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:30.080358 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:30.080659 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:30.080966 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 14:50:30.081184 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7420,"thread_start_us":7337,"threads_started":1}
I20260619 14:50:30.082159 8758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:30.177620 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 8627
I20260619 14:50:30.196671 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 8767
Jun 19 14:50:30 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.241.2@KRBTEST.COM
I20260619 14:50:30.210712 9060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:30.197487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.0.241.2:38643)
0619 14:50:30.197686 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:30.197692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:30.197714 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:50:30.197913 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:30.197918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:30.197988 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:30.198103 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:30.199410 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:30.200130 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:30.201379 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:30.201792 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:30.201928 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:30.205937 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:30.205949 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:30.205964 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:30.205995 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:30.208570 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:30.209143 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:30.209149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:30.209151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:30.209277 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:30.209565 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:30.209568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:30.209569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:30.209881 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:30.210093 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:30.210332 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 14:50:30.210476 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:50:30.211308 8897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:30.354377 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 8767
I20260619 14:50:30.379706 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 8906
Jun 19 14:50:30 dist-test-slave-p3ml krb5kdc[8511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.241.3@KRBTEST.COM
I20260619 14:50:30.393685 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:30.380383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.0.241.3:32919)
0619 14:50:30.380842 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:30.380847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:30.380862 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:50:30.380912 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:30.380916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:30.380976 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:30.381064 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:30.382275 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:30.382932 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:30.384148 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:30.384522 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:30.384611 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:30.388756 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:30.388769 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:30.388784 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:30.388813 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:30.391383 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:30.392049 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:30.392056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:30.392058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:30.392171 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:30.392497 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:30.392501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:30.392503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:30.392835 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:30.393046 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:30.393264 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 14:50:30.393439 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":258,"threads_started":1}
I20260619 14:50:30.394323 9036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:30.517186 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 8906
I20260619 14:50:30.539216 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 8524
I20260619 14:50:30.540488 8596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:30.653985 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 8524
2026-06-19T14:50:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5462 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:50:30 dist-test-slave-p3ml krb5kdc[9171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:50:30 dist-test-slave-p3ml krb5kdc[9171](info): set up 2 sockets
Jun 19 14:50:30 dist-test-slave-p3ml krb5kdc[9171](info): commencing operation
krb5kdc: starting...
W20260619 14:50:31.756512 964 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.008s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:50:31 dist-test-slave-p3ml krb5kdc[9171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:50:31.966670 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:40065
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:33929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:40065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:32.282315 9184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:32.282585 9184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:32.282758 9184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:32.290096 9184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:32.290274 9184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:32.290406 9184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:32.290505 9184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:32.290598 9184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:32.300858 9184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:33929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:40065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:40065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:32.302109 9184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:32.303475 9184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:32.315618 9192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:32.315676 9190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:32.315876 9189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:32.316629 9184 server_base.cc:1034] running on GCE node
I20260619 14:50:32.317471 9184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:32.318792 9184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:32.320048 9184 hybrid_clock.cc:648] HybridClock initialized: now 1781880632319983 us; error 47 us; skew 500 ppm
I20260619 14:50:32.320410 9184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T14:50:32Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1736 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:50:32 dist-test-slave-p3ml krb5kdc[9198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:50:32 dist-test-slave-p3ml krb5kdc[9198](info): set up 2 sockets
Jun 19 14:50:32 dist-test-slave-p3ml krb5kdc[9198](info): commencing operation
krb5kdc: starting...
W20260619 14:50:33.500381 964 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.006s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:50:33 dist-test-slave-p3ml krb5kdc[9198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:50:33.713173 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:39039
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:39039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:34.025334 9211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:34.025688 9211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:34.025873 9211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:34.032994 9211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:34.033157 9211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:34.033272 9211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:34.033377 9211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:34.033447 9211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:34.043200 9211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:39039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:39039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:34.044570 9211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:34.045907 9211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:34.057497 9216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:34.057528 9217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:34.057492 9219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:34.058574 9211 server_base.cc:1034] running on GCE node
I20260619 14:50:34.059324 9211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:34.060796 9211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:34.061996 9211 hybrid_clock.cc:648] HybridClock initialized: now 1781880634061952 us; error 34 us; skew 500 ppm
I20260619 14:50:34.062384 9211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:34 dist-test-slave-p3ml krb5kdc[9198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:34.065423 9211 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:50:34.066854 9211 webserver.cc:460] Webserver started at http://127.0.241.62:41365/ using document root <none> and password file <none>
I20260619 14:50:34.067598 9211 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:34.067744 9211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:34.068058 9211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:34.070950 9211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "427d9120ab7c442d84253531028668d4"
format_stamp: "Formatted at 2026-06-19 14:50:34 on dist-test-slave-p3ml"
server_key: "63a5b6ba8a3290006a8cf587e2dbf6f6"
server_key_iv: "f7c17717c72ab94c38be5bb6ebf11c79"
server_key_version: "encryptionkey@0"
I20260619 14:50:34.071784 9211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "427d9120ab7c442d84253531028668d4"
format_stamp: "Formatted at 2026-06-19 14:50:34 on dist-test-slave-p3ml"
server_key: "63a5b6ba8a3290006a8cf587e2dbf6f6"
server_key_iv: "f7c17717c72ab94c38be5bb6ebf11c79"
server_key_version: "encryptionkey@0"
I20260619 14:50:34.076793 9211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:50:34.080229 9226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:34.081519 9211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:34.081694 9211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "427d9120ab7c442d84253531028668d4"
format_stamp: "Formatted at 2026-06-19 14:50:34 on dist-test-slave-p3ml"
server_key: "63a5b6ba8a3290006a8cf587e2dbf6f6"
server_key_iv: "f7c17717c72ab94c38be5bb6ebf11c79"
server_key_version: "encryptionkey@0"
I20260619 14:50:34.081889 9211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:34.107538 9211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:34.111919 9211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:34.112221 9211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:34.130281 9211 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:39039
I20260619 14:50:34.130316 9277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:39039 every 8 connection(s)
I20260619 14:50:34.131693 9211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:34.133270 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 9211
I20260619 14:50:34.133491 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:34.134272 964 external_mini_cluster.cc:1428] Setting key 498f9c90a018ba2a40a6dfadc8f1dcdc
I20260619 14:50:34.136468 9278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 14:50:34 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:34.149657 9278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4: Bootstrap starting.
I20260619 14:50:34.153770 9278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:34.155011 9278 log.cc:824] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:34.158213 9278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4: No bootstrap required, opened a new log
I20260619 14:50:34.158355 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:34.136298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60862 (local address 127.0.241.62:39039)
0619 14:50:34.137115 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:34.137129 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:34.137177 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:50:34.139312 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:34.139338 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:34.139852 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:34.140320 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:34.141615 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:34.143712 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:34.144913 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:34.145931 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:34.146123 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:34.150116 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:34.150126 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:34.150144 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:34.150195 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:34.154040 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:34.154688 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:34.154696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:34.154704 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:34.154900 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:34.155238 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:34.155243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:34.155246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:34.155932 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:34.156155 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:34.156691 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 14:50:34.157084 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":383,"threads_started":1}
I20260619 14:50:34.166183 9278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427d9120ab7c442d84253531028668d4" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39039 } }
I20260619 14:50:34.166522 9278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:34.166594 9278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427d9120ab7c442d84253531028668d4, State: Initialized, Role: FOLLOWER
I20260619 14:50:34.167136 9278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427d9120ab7c442d84253531028668d4" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39039 } }
I20260619 14:50:34.167328 9278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:34.167419 9278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:34.167603 9278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:34.169617 9278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427d9120ab7c442d84253531028668d4" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39039 } }
I20260619 14:50:34.170193 9278 leader_election.cc:302] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 427d9120ab7c442d84253531028668d4; no voters:
I20260619 14:50:34.170790 9278 leader_election.cc:288] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:34.170917 9283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:34.171617 9283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 1 LEADER]: Becoming Leader. State: Replica: 427d9120ab7c442d84253531028668d4, State: Running, Role: LEADER
I20260619 14:50:34.172188 9283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427d9120ab7c442d84253531028668d4" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39039 } }
I20260619 14:50:34.172884 9278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:34.174705 9284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "427d9120ab7c442d84253531028668d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427d9120ab7c442d84253531028668d4" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39039 } } }
I20260619 14:50:34.175021 9284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:34.175984 9285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 427d9120ab7c442d84253531028668d4. Latest consensus state: current_term: 1 leader_uuid: "427d9120ab7c442d84253531028668d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427d9120ab7c442d84253531028668d4" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39039 } } }
I20260619 14:50:34.177649 9285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:34.178053 9289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:34.183063 9289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:34.192377 9289 catalog_manager.cc:1347] Generated new cluster ID: aae8182c4c234eada3165d363982c74a
I20260619 14:50:34.192507 9289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:34.209003 9289 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:34.210253 9289 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:34.219106 9289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4: Generated new TSK 0
I20260619 14:50:34.219995 9289 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:50:34.281002 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:39039
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:34.607030 9304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:34.607302 9304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:34.607498 9304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:34.614722 9304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:34.614825 9304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:34.614995 9304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:50:34.624536 9304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:34.625761 9304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:34.627081 9304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:34.639917 9310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:34.639936 9312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:34.639981 9309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:34.640396 9304 server_base.cc:1034] running on GCE node
I20260619 14:50:34.641048 9304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:34.641942 9304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:34.643175 9304 hybrid_clock.cc:648] HybridClock initialized: now 1781880634643141 us; error 52 us; skew 500 ppm
I20260619 14:50:34.643548 9304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:34 dist-test-slave-p3ml krb5kdc[9198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:34.646591 9304 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:50:34.648034 9304 webserver.cc:460] Webserver started at http://127.0.241.1:44633/ using document root <none> and password file <none>
I20260619 14:50:34.648638 9304 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:34.648733 9304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:34.648994 9304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:34.651836 9304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "ca07ee25b71640d280f50758dfee497a"
format_stamp: "Formatted at 2026-06-19 14:50:34 on dist-test-slave-p3ml"
server_key: "b1db3d3d6f2782a57749f358953b0fc6"
server_key_iv: "241e3a77b14f24743674f153f299b4a8"
server_key_version: "encryptionkey@0"
I20260619 14:50:34.652545 9304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "ca07ee25b71640d280f50758dfee497a"
format_stamp: "Formatted at 2026-06-19 14:50:34 on dist-test-slave-p3ml"
server_key: "b1db3d3d6f2782a57749f358953b0fc6"
server_key_iv: "241e3a77b14f24743674f153f299b4a8"
server_key_version: "encryptionkey@0"
I20260619 14:50:34.657845 9304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:50:34.661211 9319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:34.662534 9304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:50:34.662667 9304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "ca07ee25b71640d280f50758dfee497a"
format_stamp: "Formatted at 2026-06-19 14:50:34 on dist-test-slave-p3ml"
server_key: "b1db3d3d6f2782a57749f358953b0fc6"
server_key_iv: "241e3a77b14f24743674f153f299b4a8"
server_key_version: "encryptionkey@0"
I20260619 14:50:34.662842 9304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:34.673084 9304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:34.676833 9304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:34.677057 9304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:34.677906 9304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:34.679206 9304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:34.679276 9304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:34.679356 9304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:34.679400 9304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:34.711611 9304 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:46243
I20260619 14:50:34.711643 9431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:46243 every 8 connection(s)
I20260619 14:50:34.712881 9304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:34.714960 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 9304
I20260619 14:50:34.715158 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:34.715828 964 external_mini_cluster.cc:1428] Setting key 9bf11717450da88f5d63d972bf1125ec
Jun 19 14:50:34 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
I20260619 14:50:34.739354 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:34.716081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:37305 (local address 127.0.241.62:39039)
0619 14:50:34.716584 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:34.716591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:34.717079 (+ 488us) server_negotiation.cc:407] Connection header received
0619 14:50:34.719115 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:34.719124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:34.719210 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:34.719349 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:34.720826 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:34.721957 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:34.723659 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:34.724412 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:34.724559 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:34.733767 (+ 9208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:34.733772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:34.733775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:34.733808 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:34.736360 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:34.737172 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:34.737177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:34.737179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:34.737315 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:34.737780 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:34.737783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:34.737785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:34.738049 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:34.738158 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:34.738948 (+ 790us) server_negotiation.cc:299] Negotiation successful
0619 14:50:34.739113 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":216,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
I20260619 14:50:34.740703 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:34.716005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:37305)
0619 14:50:34.716889 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:34.716966 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:34.718860 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:34.719619 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:34.719627 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:34.720106 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:34.720654 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:34.720671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:34.722142 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:34.722146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:34.723521 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:34.723539 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:34.724610 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:34.724615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:34.724994 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:34.730790 (+ 5796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:34.730825 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:34.733500 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:34.736507 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:34.736518 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:34.736535 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:34.737033 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:34.737444 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:34.737451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:34.737456 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:34.737669 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:34.738194 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:34.738201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:34.738617 (+ 416us) client_negotiation.cc:769] Sending connection context
0619 14:50:34.738848 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 14:50:34.739313 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":430,"threads_started":1}
I20260619 14:50:34.742275 9432 heartbeater.cc:347] Connected to a master server at 127.0.241.62:39039
I20260619 14:50:34.742600 9432 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:34.743307 9432 heartbeater.cc:511] Master 127.0.241.62:39039 requested a full tablet report, sending...
I20260619 14:50:34.745350 9243 ts_manager.cc:194] Registered new tserver with Master: ca07ee25b71640d280f50758dfee497a (127.0.241.1:46243)
I20260619 14:50:34.747394 9243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:37305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:50:34.766664 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:39039
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:35.083266 9438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:35.083552 9438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:35.083724 9438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:35.091380 9438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:35.091564 9438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:35.091781 9438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:50:35.101328 9438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:35.102615 9438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:35.103982 9438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:35.116997 9446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:35.117055 9444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:35.117081 9443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:35.117930 9438 server_base.cc:1034] running on GCE node
I20260619 14:50:35.118469 9438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:35.119287 9438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:35.120482 9438 hybrid_clock.cc:648] HybridClock initialized: now 1781880635120427 us; error 64 us; skew 500 ppm
I20260619 14:50:35.120798 9438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:35 dist-test-slave-p3ml krb5kdc[9198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:35.123761 9438 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:50:35.125103 9438 webserver.cc:460] Webserver started at http://127.0.241.2:43195/ using document root <none> and password file <none>
I20260619 14:50:35.125762 9438 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:35.125910 9438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:35.126227 9438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:35.129029 9438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "41b941bfc4e64417a6ffbb3d9f137f54"
format_stamp: "Formatted at 2026-06-19 14:50:35 on dist-test-slave-p3ml"
server_key: "bd7c0b6e0ae729a20cac8673f6a49079"
server_key_iv: "5e04a8e2bb5b4412e880e64c53449b4a"
server_key_version: "encryptionkey@0"
I20260619 14:50:35.129815 9438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "41b941bfc4e64417a6ffbb3d9f137f54"
format_stamp: "Formatted at 2026-06-19 14:50:35 on dist-test-slave-p3ml"
server_key: "bd7c0b6e0ae729a20cac8673f6a49079"
server_key_iv: "5e04a8e2bb5b4412e880e64c53449b4a"
server_key_version: "encryptionkey@0"
I20260619 14:50:35.134886 9438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 14:50:35.138550 9453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:35.139976 9438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:35.140154 9438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "41b941bfc4e64417a6ffbb3d9f137f54"
format_stamp: "Formatted at 2026-06-19 14:50:35 on dist-test-slave-p3ml"
server_key: "bd7c0b6e0ae729a20cac8673f6a49079"
server_key_iv: "5e04a8e2bb5b4412e880e64c53449b4a"
server_key_version: "encryptionkey@0"
I20260619 14:50:35.140352 9438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:35.160331 9438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:35.164127 9438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:35.164414 9438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:35.165329 9438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:35.166626 9438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:35.166734 9438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:35.166873 9438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:35.166949 9438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:35.199474 9438 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:39059
I20260619 14:50:35.199569 9565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:39059 every 8 connection(s)
I20260619 14:50:35.200731 9438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:35.207679 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 9438
I20260619 14:50:35.207866 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:35.208353 964 external_mini_cluster.cc:1428] Setting key 9756214420cd03882686ac59dc8eba53
Jun 19 14:50:35 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:35.220166 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:35.203122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:56023 (local address 127.0.241.62:39039)
0619 14:50:35.203361 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:35.203367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:35.204571 (+ 1204us) server_negotiation.cc:407] Connection header received
0619 14:50:35.206266 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:35.206270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:35.206334 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:35.206419 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:35.208120 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:35.208887 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:35.210709 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:35.211093 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:35.211215 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:35.214603 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:35.214608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:35.214611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:35.214641 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:35.217181 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:35.217885 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:35.217894 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:35.217897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:35.218017 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:35.218445 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:35.218449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:35.218452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:35.218714 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:35.218872 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:35.219596 (+ 724us) server_negotiation.cc:299] Negotiation successful
0619 14:50:35.219804 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 14:50:35.221056 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:35.203701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:56023)
0619 14:50:35.204395 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:35.204431 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:35.206018 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:35.206626 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:35.206634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:35.207139 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:35.207938 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:35.207963 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:35.209040 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:35.209047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:35.210555 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:35.210566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:35.211315 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:35.211322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:35.211626 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:35.212298 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:35.212322 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:35.214376 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:35.217317 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:35.217326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:35.217342 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:35.217760 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:35.218143 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:35.218148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:35.218152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:35.218329 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:35.218846 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:35.218854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:35.219182 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 14:50:35.219498 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 14:50:35.219986 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":276,"threads_started":1}
I20260619 14:50:35.222467 9566 heartbeater.cc:347] Connected to a master server at 127.0.241.62:39039
I20260619 14:50:35.222784 9566 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:35.223701 9566 heartbeater.cc:511] Master 127.0.241.62:39039 requested a full tablet report, sending...
I20260619 14:50:35.225337 9243 ts_manager.cc:194] Registered new tserver with Master: 41b941bfc4e64417a6ffbb3d9f137f54 (127.0.241.2:39059)
I20260619 14:50:35.226570 9243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:56023
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:50:35.258073 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:39039
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:35.581508 9571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:35.581779 9571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:35.581949 9571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:35.589653 9571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:35.589792 9571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:35.589977 9571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:50:35.599761 9571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:35.601078 9571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:35.602360 9571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:35.615217 9579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:35.615260 9576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:35.615281 9577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:35.616185 9571 server_base.cc:1034] running on GCE node
I20260619 14:50:35.616780 9571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:35.617673 9571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:35.618888 9571 hybrid_clock.cc:648] HybridClock initialized: now 1781880635618835 us; error 64 us; skew 500 ppm
I20260619 14:50:35.619192 9571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:35 dist-test-slave-p3ml krb5kdc[9198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:35.622294 9571 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:50:35.623730 9571 webserver.cc:460] Webserver started at http://127.0.241.3:40003/ using document root <none> and password file <none>
I20260619 14:50:35.624404 9571 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:35.624557 9571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:35.624861 9571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:35.627766 9571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "981467fc430749d2941503aebdea893a"
format_stamp: "Formatted at 2026-06-19 14:50:35 on dist-test-slave-p3ml"
server_key: "b8ac4401422e7c94eeee90c8651cf426"
server_key_iv: "70d738d547f102656accd4597b0e874a"
server_key_version: "encryptionkey@0"
I20260619 14:50:35.628567 9571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "981467fc430749d2941503aebdea893a"
format_stamp: "Formatted at 2026-06-19 14:50:35 on dist-test-slave-p3ml"
server_key: "b8ac4401422e7c94eeee90c8651cf426"
server_key_iv: "70d738d547f102656accd4597b0e874a"
server_key_version: "encryptionkey@0"
I20260619 14:50:35.633689 9571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:50:35.637324 9586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:35.639065 9571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:50:35.639235 9571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "981467fc430749d2941503aebdea893a"
format_stamp: "Formatted at 2026-06-19 14:50:35 on dist-test-slave-p3ml"
server_key: "b8ac4401422e7c94eeee90c8651cf426"
server_key_iv: "70d738d547f102656accd4597b0e874a"
server_key_version: "encryptionkey@0"
I20260619 14:50:35.639492 9571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:35.652817 9571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:35.656744 9571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:35.657042 9571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:35.657960 9571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:35.659314 9571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:35.659416 9571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:35.659588 9571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:35.659665 9571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:35.695281 9571 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:43671
I20260619 14:50:35.695322 9698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:43671 every 8 connection(s)
I20260619 14:50:35.696566 9571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:35.700037 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 9571
I20260619 14:50:35.700219 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:35.700736 964 external_mini_cluster.cc:1428] Setting key 92866e2b680456bec4c4bae24f36de0c
Jun 19 14:50:35 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:35.716361 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:35.699043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:59759 (local address 127.0.241.62:39039)
0619 14:50:35.699205 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:35.699209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:35.700511 (+ 1302us) server_negotiation.cc:407] Connection header received
0619 14:50:35.702458 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:35.702463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:35.702553 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:35.702663 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:35.704209 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:35.704950 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:35.706505 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:35.706907 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:35.707058 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:35.710340 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:35.710344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:35.710347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:35.710377 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:35.712963 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:35.713814 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:35.713820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:35.713823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:35.713959 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:35.714494 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:35.714499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:35.714502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:35.714834 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:35.715012 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:35.715956 (+ 944us) server_negotiation.cc:299] Negotiation successful
0619 14:50:35.716115 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:50:35.717419 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:35.699425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:59759)
0619 14:50:35.700326 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:35.700372 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:35.702175 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:35.702912 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:35.702920 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:35.703421 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:35.704057 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:35.704074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:35.705100 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:35.705105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:35.706358 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:35.706369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:35.707161 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:35.707168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:35.707390 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:35.708048 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:35.708069 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:35.710165 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:35.713104 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:35.713112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:35.713127 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:35.713680 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:35.714102 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:35.714106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:35.714122 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:35.714365 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:35.715116 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:35.715123 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:35.715554 (+ 431us) client_negotiation.cc:769] Sending connection context
0619 14:50:35.715843 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 14:50:35.716343 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":349,"threads_started":1}
I20260619 14:50:35.718602 9699 heartbeater.cc:347] Connected to a master server at 127.0.241.62:39039
I20260619 14:50:35.718853 9699 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:35.719534 9699 heartbeater.cc:511] Master 127.0.241.62:39039 requested a full tablet report, sending...
I20260619 14:50:35.720749 9243 ts_manager.cc:194] Registered new tserver with Master: 981467fc430749d2941503aebdea893a (127.0.241.3:43671)
I20260619 14:50:35.721464 9243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:59759
I20260619 14:50:35.728433 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.1@KRBTEST.COM" created.
I20260619 14:50:35.750478 9432 heartbeater.cc:503] Master 127.0.241.62:39039 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab.
Entry for principal oryx/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab.
WARNING: no policy specified for oryx/127.0.241.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab.
Entry for principal oryx/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab.
WARNING: no policy specified for oryx/127.0.241.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab.
Entry for principal oryx/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab.
I20260619 14:50:35.861620 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 9304
Jun 19 14:50:35 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:50:35.877569 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:35.862862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33002 (local address 127.0.241.1:46243)
0619 14:50:35.863615 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:35.863622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:35.863647 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:50:35.863707 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:35.863714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:35.863924 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:35.864068 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:35.865302 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:35.866399 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:35.867683 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:35.868238 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:35.868334 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:35.872113 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:35.872124 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:35.872137 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:35.872173 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:35.875076 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:35.875738 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:35.875745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:35.875747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:35.875872 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:35.876175 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:35.876178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:35.876180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:35.876506 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:35.876714 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:35.876956 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 14:50:35.877140 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":396,"threads_started":1}
I20260619 14:50:35.878878 9427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:35.960775 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 9304
I20260619 14:50:35.976982 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 9438
Jun 19 14:50:35 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:50:35.992547 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:35.977854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.0.241.2:39059)
0619 14:50:35.978399 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:35.978405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:35.978430 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:50:35.978498 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:35.978504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:35.978737 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:35.978861 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:35.980339 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:35.981311 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:35.982498 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:35.983029 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:35.983122 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:35.987530 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:35.987541 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:35.987556 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:35.987596 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:35.990071 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:35.990657 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:35.990663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:35.990666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:35.990775 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:35.991057 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:35.991060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:35.991062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:35.991362 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:35.991691 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:35.991957 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 14:50:35.992141 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":282,"threads_started":1}
I20260619 14:50:35.993750 9561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:36.079172 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 9438
I20260619 14:50:36.095019 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 9571
Jun 19 14:50:36 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:50:36.111334 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:36.096063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.0.241.3:43671)
0619 14:50:36.096669 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:36.096675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:36.096702 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:50:36.096756 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:36.096762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:36.096963 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:36.097139 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:36.098414 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:36.099374 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:36.100757 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:36.101320 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:36.101469 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:36.105777 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:36.105789 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:36.105804 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:36.105837 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:36.108593 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:36.109269 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:36.109276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:36.109278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:36.109399 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:36.109781 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:36.109788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:36.109791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:36.110123 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:36.110441 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:36.110721 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 14:50:36.110981 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":317,"threads_started":1}
I20260619 14:50:36.112605 9694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:36.194058 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 9571
I20260619 14:50:36.209424 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 9211
I20260619 14:50:36.210640 9273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:36.310746 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 9211
I20260619 14:50:36.330850 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:39039
--webserver_interface=127.0.241.62
--webserver_port=41365
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:39039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:36.648386 9716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:36.648656 9716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:36.648846 9716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:36.656019 9716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:36.656152 9716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:36.656237 9716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:36.656347 9716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:36.656435 9716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:36.666102 9716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:39039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:39039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=41365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:36.667424 9716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:36.668808 9716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:36.680174 9722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:36.680305 9721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:36.680366 9724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:36.681540 9716 server_base.cc:1034] running on GCE node
I20260619 14:50:36.682281 9716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:36.683652 9716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:36.684883 9716 hybrid_clock.cc:648] HybridClock initialized: now 1781880636684830 us; error 59 us; skew 500 ppm
I20260619 14:50:36.685236 9716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:36 dist-test-slave-p3ml krb5kdc[9198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:36.688349 9716 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:50:36.689744 9716 webserver.cc:460] Webserver started at http://127.0.241.62:41365/ using document root <none> and password file <none>
I20260619 14:50:36.690416 9716 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:36.690577 9716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:36.695950 9716 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:36.698683 9731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:36.700119 9716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:36.700284 9716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "427d9120ab7c442d84253531028668d4"
format_stamp: "Formatted at 2026-06-19 14:50:34 on dist-test-slave-p3ml"
server_key: "63a5b6ba8a3290006a8cf587e2dbf6f6"
server_key_iv: "f7c17717c72ab94c38be5bb6ebf11c79"
server_key_version: "encryptionkey@0"
I20260619 14:50:36.701174 9716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:36.719298 9716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:36.723114 9716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:36.723369 9716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:36.741262 9716 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:39039
I20260619 14:50:36.741299 9782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:39039 every 8 connection(s)
I20260619 14:50:36.742803 9716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:36.749475 9783 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:50:36.752504 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 9716
I20260619 14:50:36.752595 9783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4: Bootstrap starting.
I20260619 14:50:36.753305 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:46243
--local_ip_for_outbound_sockets=127.0.241.1
--tserver_master_addrs=127.0.241.62:39039
--webserver_port=44633
--webserver_interface=127.0.241.1
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.241.1
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:50:36.770865 9783 log.cc:824] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:36.779325 9783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:50:36.779889 9783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4: Bootstrap complete.
I20260619 14:50:36.788820 9783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427d9120ab7c442d84253531028668d4" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39039 } }
I20260619 14:50:36.789641 9783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427d9120ab7c442d84253531028668d4, State: Initialized, Role: FOLLOWER
I20260619 14:50:36.790279 9783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427d9120ab7c442d84253531028668d4" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39039 } }
I20260619 14:50:36.790520 9783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:36.790637 9783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:36.790817 9783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:50:36.794412 9783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427d9120ab7c442d84253531028668d4" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39039 } }
I20260619 14:50:36.794870 9783 leader_election.cc:302] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 427d9120ab7c442d84253531028668d4; no voters:
I20260619 14:50:36.795595 9783 leader_election.cc:288] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:50:36.795678 9787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:50:36.796994 9787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [term 2 LEADER]: Becoming Leader. State: Replica: 427d9120ab7c442d84253531028668d4, State: Running, Role: LEADER
I20260619 14:50:36.797526 9787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427d9120ab7c442d84253531028668d4" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39039 } }
I20260619 14:50:36.797765 9783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:36.800287 9789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 427d9120ab7c442d84253531028668d4. Latest consensus state: current_term: 2 leader_uuid: "427d9120ab7c442d84253531028668d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427d9120ab7c442d84253531028668d4" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39039 } } }
I20260619 14:50:36.800401 9788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "427d9120ab7c442d84253531028668d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427d9120ab7c442d84253531028668d4" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 39039 } } }
I20260619 14:50:36.800727 9789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:36.800812 9788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:36.810452 9794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:36.816661 9794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:36.818081 9794 catalog_manager.cc:1259] Loaded cluster ID: aae8182c4c234eada3165d363982c74a
I20260619 14:50:36.818182 9794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:36.821897 9794 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:36.823261 9794 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 427d9120ab7c442d84253531028668d4: Loaded TSK: 0
I20260619 14:50:36.824800 9794 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:50:37.085008 9785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:37.085263 9785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:37.085415 9785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:37.092770 9785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:37.092872 9785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:37.093040 9785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:50:37.103016 9785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/oryx_127.0.241.1.keytab
--principal=oryx/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:46243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=44633
--tserver_master_addrs=127.0.241.62:39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:37.104413 9785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:37.105854 9785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:37.120149 9809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:37.120180 9810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:37.120644 9812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:37.121264 9785 server_base.cc:1034] running on GCE node
I20260619 14:50:37.121879 9785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:37.122910 9785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:37.124114 9785 hybrid_clock.cc:648] HybridClock initialized: now 1781880637124067 us; error 48 us; skew 500 ppm
I20260619 14:50:37.124410 9785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880637, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:37.127703 9785 init.cc:377] Logged in from keytab as oryx/127.0.241.1@KRBTEST.COM (short username oryx)
I20260619 14:50:37.129236 9785 webserver.cc:460] Webserver started at http://127.0.241.1:44633/ using document root <none> and password file <none>
I20260619 14:50:37.129889 9785 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:37.130043 9785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:37.135676 9785 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:50:37.138471 9819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:37.139853 9785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:37.140025 9785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "ca07ee25b71640d280f50758dfee497a"
format_stamp: "Formatted at 2026-06-19 14:50:34 on dist-test-slave-p3ml"
server_key: "b1db3d3d6f2782a57749f358953b0fc6"
server_key_iv: "241e3a77b14f24743674f153f299b4a8"
server_key_version: "encryptionkey@0"
I20260619 14:50:37.140913 9785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:37.157727 9785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:37.161717 9785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:37.161993 9785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:37.162907 9785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:37.164273 9785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:37.164393 9785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:37.164528 9785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:37.164621 9785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:37.197317 9785 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:46243
I20260619 14:50:37.197383 9931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:46243 every 8 connection(s)
I20260619 14:50:37.198760 9785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:37.204372 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 9785
I20260619 14:50:37.205133 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:39059
--local_ip_for_outbound_sockets=127.0.241.2
--tserver_master_addrs=127.0.241.62:39039
--webserver_port=43195
--webserver_interface=127.0.241.2
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.241.2
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:50:37.220973 9934 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:37.201753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:41149 (local address 127.0.241.62:39039)
0619 14:50:37.202633 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:37.202652 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:37.202927 (+ 275us) server_negotiation.cc:407] Connection header received
0619 14:50:37.206006 (+ 3079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:37.206028 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:37.206527 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:37.206967 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:37.208523 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.210620 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.212819 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.213494 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.213639 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:37.220054 (+ 6415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:41149: BlockingRecv error: recv got EOF from 127.0.241.1:41149 (error 108)
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":361,"threads_started":1}
I20260619 14:50:37.220988 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:37.201825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:41149)
0619 14:50:37.202700 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:37.202757 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:37.205178 (+ 2421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:37.207133 (+ 1955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:37.207141 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:37.207749 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:37.208306 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:37.208322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.210791 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.210794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:37.212633 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:37.212653 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.213739 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.213748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:37.214071 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:37.214821 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:37.214854 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:37.219707 (+ 4853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":382,"threads_started":1}
W20260619 14:50:37.221902 9932 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:39039 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.241.62:39039: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:37.231145 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:37.222582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:37481)
0619 14:50:37.222758 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:37.222779 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:37.222910 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:37.223327 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:37.223335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:37.223693 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:37.223856 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:37.223864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.225469 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.225473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:37.226412 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:37.226425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.227522 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.227526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:37.227640 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:37.228253 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:37.228270 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:37.230952 (+ 2682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:50:37.231390 9934 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:37.222774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:37481 (local address 127.0.241.62:39039)
0619 14:50:37.222997 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:37.223005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:37.223025 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:50:37.223080 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:37.223087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:37.223171 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:37.223282 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:37.223997 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.225313 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.226569 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.227333 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.227514 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:37.231177 (+ 3663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:37481: BlockingRecv error: recv got EOF from 127.0.241.1:37481 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:37.239255 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:37.231980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:45881)
0619 14:50:37.232136 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:37.232154 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:37.232265 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:37.232599 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:37.232603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:37.232901 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:37.233022 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:37.233029 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.234077 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.234083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:37.234993 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:37.235003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.235711 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.235716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:37.235935 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:37.236500 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:37.236515 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:37.238983 (+ 2468us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 14:50:37.239555 9932 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:50:37.239703 9934 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:37.232171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:45881 (local address 127.0.241.62:39039)
0619 14:50:37.232342 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:37.232348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:37.232365 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:50:37.232413 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:37.232416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:37.232480 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:37.232580 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:37.233168 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.233945 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.235137 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.235579 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.235703 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:37.239151 (+ 3448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:45881: BlockingRecv error: recv got EOF from 127.0.241.1:45881 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 14:50:37.533195 9936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:37.533504 9936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:37.533676 9936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:37.541018 9936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:37.541170 9936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:37.541338 9936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:50:37.551400 9936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/oryx_127.0.241.2.keytab
--principal=oryx/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:39059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=43195
--tserver_master_addrs=127.0.241.62:39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:37.552724 9936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:37.554013 9936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:37.566840 9944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:37.566922 9941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:37.567032 9942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:37.567999 9936 server_base.cc:1034] running on GCE node
I20260619 14:50:37.568799 9936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:37.570777 9936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:37.571993 9936 hybrid_clock.cc:648] HybridClock initialized: now 1781880637571929 us; error 54 us; skew 500 ppm
I20260619 14:50:37.572402 9936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880637, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:37.575937 9936 init.cc:377] Logged in from keytab as oryx/127.0.241.2@KRBTEST.COM (short username oryx)
I20260619 14:50:37.577301 9936 webserver.cc:460] Webserver started at http://127.0.241.2:43195/ using document root <none> and password file <none>
I20260619 14:50:37.578014 9936 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:37.578174 9936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:37.583385 9936 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260619 14:50:37.585994 9951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:37.587194 9936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:50:37.587373 9936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "41b941bfc4e64417a6ffbb3d9f137f54"
format_stamp: "Formatted at 2026-06-19 14:50:35 on dist-test-slave-p3ml"
server_key: "bd7c0b6e0ae729a20cac8673f6a49079"
server_key_iv: "5e04a8e2bb5b4412e880e64c53449b4a"
server_key_version: "encryptionkey@0"
I20260619 14:50:37.588423 9936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:37.612036 9936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:37.616261 9936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:37.616557 9936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:37.617496 9936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:37.618772 9936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:37.618898 9936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:37.619024 9936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:37.619114 9936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:37.651713 9936 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:39059
I20260619 14:50:37.651799 10063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:39059 every 8 connection(s)
I20260619 14:50:37.653154 9936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:37.660888 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 9936
I20260619 14:50:37.661469 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:43671
--local_ip_for_outbound_sockets=127.0.241.3
--tserver_master_addrs=127.0.241.62:39039
--webserver_port=40003
--webserver_interface=127.0.241.3
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.241.3
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:50:37.668558 9934 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:37.655739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:33451 (local address 127.0.241.62:39039)
0619 14:50:37.655919 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:37.655923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:37.657044 (+ 1121us) server_negotiation.cc:407] Connection header received
0619 14:50:37.658836 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:37.658843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:37.658933 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:37.659050 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:37.660512 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.661290 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.662886 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.663305 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.663438 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:37.668241 (+ 4803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:33451: BlockingRecv error: recv got EOF from 127.0.241.2:33451 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:50:37.669037 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:37.656211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:33451)
0619 14:50:37.656892 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:37.656928 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:37.658581 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:37.659278 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:37.659285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:37.659827 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:37.660347 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:37.660362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.661441 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.661459 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:37.662743 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:37.662754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.663551 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.663558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:37.663766 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:37.664440 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:37.664460 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:37.667990 (+ 3530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":287,"threads_started":1}
W20260619 14:50:37.669729 10064 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:39039 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.241.62:39039: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:37.677762 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:37.670282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:58511)
0619 14:50:37.670454 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:37.670473 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:37.670567 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:37.670877 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:37.670881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:37.671196 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:37.671319 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:37.671326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.672312 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.672317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:37.673188 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:37.673198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.673873 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.673879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:37.674058 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:37.674659 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:37.674675 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:37.677488 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 14:50:37.678053 9934 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:37.670378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:58511 (local address 127.0.241.62:39039)
0619 14:50:37.670543 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:37.670548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:37.670566 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:50:37.670682 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:37.670687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:37.670754 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:37.670866 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:37.671471 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.672177 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.673325 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.673727 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.673872 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:37.677670 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:58511: BlockingRecv error: recv got EOF from 127.0.241.2:58511 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:37 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:37.685911 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:37.678385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:43849)
0619 14:50:37.678520 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:37.678538 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:37.678634 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:37.679041 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:37.679046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:37.679324 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:37.679523 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:37.679532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.680444 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.680449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:37.681325 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:37.681336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.682017 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.682022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:37.682187 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:37.682786 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:37.682801 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:37.685638 (+ 2837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 14:50:37.686295 10064 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:50:37.686415 9934 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:37.678555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:43849 (local address 127.0.241.62:39039)
0619 14:50:37.678814 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:37.678817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:37.678830 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:50:37.678871 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:37.678874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:37.678920 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:37.679021 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:37.679647 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.680310 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.681474 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:37.681880 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:37.682008 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:37.685904 (+ 3896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:43849: BlockingRecv error: recv got EOF from 127.0.241.2:43849 (error 108)
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":134}
W20260619 14:50:37.982798 10067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:37.983101 10067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:37.983263 10067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:37.990556 10067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:37.990694 10067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:37.990878 10067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:50:38.000614 10067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/krb5kdc/oryx_127.0.241.3.keytab
--principal=oryx/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:43671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=40003
--tserver_master_addrs=127.0.241.62:39039
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:38.001827 10067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:38.003093 10067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:38.016153 10075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:38.016260 10073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:38.016215 10072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:38.017063 10067 server_base.cc:1034] running on GCE node
I20260619 14:50:38.017606 10067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:38.018436 10067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:38.019650 10067 hybrid_clock.cc:648] HybridClock initialized: now 1781880638019619 us; error 46 us; skew 500 ppm
I20260619 14:50:38.019932 10067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:38 dist-test-slave-p3ml krb5kdc[9198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880638, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:38.022913 10067 init.cc:377] Logged in from keytab as oryx/127.0.241.3@KRBTEST.COM (short username oryx)
I20260619 14:50:38.024374 10067 webserver.cc:460] Webserver started at http://127.0.241.3:40003/ using document root <none> and password file <none>
I20260619 14:50:38.025008 10067 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:38.025157 10067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:38.030575 10067 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 14:50:38.033375 10082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:38.034636 10067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:50:38.034807 10067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "981467fc430749d2941503aebdea893a"
format_stamp: "Formatted at 2026-06-19 14:50:35 on dist-test-slave-p3ml"
server_key: "b8ac4401422e7c94eeee90c8651cf426"
server_key_iv: "70d738d547f102656accd4597b0e874a"
server_key_version: "encryptionkey@0"
I20260619 14:50:38.035810 10067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:38.049608 10067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:38.054076 10067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:38.054363 10067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:38.055327 10067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:38.056664 10067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:38.056787 10067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:38.056922 10067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:38.057011 10067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:38.089726 10067 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:43671
I20260619 14:50:38.089816 10194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:43671 every 8 connection(s)
I20260619 14:50:38.091128 10067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:38.092475 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 10067
Jun 19 14:50:38 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:38 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:50:38.108376 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:38.093936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:59117 (local address 127.0.241.62:39039)
0619 14:50:38.094392 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:38.094397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:38.095484 (+ 1087us) server_negotiation.cc:407] Connection header received
0619 14:50:38.098072 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:38.098075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:38.098181 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:38.098279 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:38.099999 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.100674 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.102725 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.103119 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.103203 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:38.108146 (+ 4943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:59117: BlockingRecv error: recv got EOF from 127.0.241.3:59117 (error 108)
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":250,"threads_started":1}
I20260619 14:50:38.109220 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:38.094504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:59117)
0619 14:50:38.095323 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:38.095371 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:38.097786 (+ 2415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:38.098568 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:38.098579 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:38.099097 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:38.099837 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:38.099860 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.100825 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.100833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:38.102574 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:38.102595 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.103334 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.103343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:38.103706 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:38.104337 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:38.104363 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:38.107981 (+ 3618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":353,"threads_started":1}
W20260619 14:50:38.109905 10195 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:39039 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.241.62:39039: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
I20260619 14:50:38.111202 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:38.093497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.0.241.62:39039)
0619 14:50:38.093723 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:38.093733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:38.096260 (+ 2527us) server_negotiation.cc:407] Connection header received
0619 14:50:38.096432 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:38.096436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:38.096527 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:38.096613 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:38.098686 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.099558 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.100897 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.101283 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.101365 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:38.104463 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:38.104471 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:38.104484 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:38.104522 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:38.107610 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:38.108418 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:38.108429 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:38.108436 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:38.108702 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:38.109084 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:38.109087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:38.109089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:38.109579 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:38.109912 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:38.110365 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 14:50:38.110816 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 14:50:38 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:38 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:38.118288 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:38.110507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:40933)
0619 14:50:38.110839 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:38.110852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:38.110935 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:38.111225 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:38.111246 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:38.111642 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:38.111841 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:38.111850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.112978 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.112983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:38.114167 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:38.114185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.114982 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.114986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:38.115102 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:38.115621 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:38.115637 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:38.118113 (+ 2476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249}
W20260619 14:50:38.118525 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:38.110568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:40933 (local address 127.0.241.62:39039)
0619 14:50:38.110741 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:38.110745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:38.110927 (+ 182us) server_negotiation.cc:407] Connection header received
0619 14:50:38.111018 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:38.111023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:38.111087 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:38.111192 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:38.111955 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.112838 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.114312 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.114861 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.114949 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:38.118280 (+ 3331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:40933: BlockingRecv error: recv got EOF from 127.0.241.3:40933 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 14:50:38 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:38 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:38.125641 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:38.118982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:50443)
0619 14:50:38.119130 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:38.119145 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:38.119215 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:38.119495 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:38.119498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:38.119748 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:38.119880 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:38.119887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.120753 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.120756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:38.121621 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:38.121631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.122241 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.122245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:38.122381 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:38.122843 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:38.122858 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:38.125388 (+ 2530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 14:50:38.125841 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:38.119031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:50443 (local address 127.0.241.62:39039)
0619 14:50:38.119230 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:38.119234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:38.119247 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:50:38.119282 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:38.119285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:38.119339 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:38.119413 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:38.119989 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.120635 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.121728 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.122135 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.122219 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:38.125572 (+ 3353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:50443: BlockingRecv error: recv got EOF from 127.0.241.3:50443 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260619 14:50:38.125941 10195 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 14:50:38 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:38 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:38.248162 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:38.240134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:42235)
0619 14:50:38.240709 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:38.240728 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:38.240834 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:38.241112 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:38.241115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:38.241535 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:38.241733 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:38.241741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.242617 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.242620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:38.243629 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:38.243641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.244496 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.244503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:38.244683 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:38.245254 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:38.245270 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:38.247944 (+ 2674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":350,"threads_started":1}
W20260619 14:50:38.248466 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:38.240194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:42235 (local address 127.0.241.62:39039)
0619 14:50:38.240365 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:38.240372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:38.240793 (+ 421us) server_negotiation.cc:407] Connection header received
0619 14:50:38.240920 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:38.240924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:38.240991 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:38.241069 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:38.241841 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.242500 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.243784 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.244319 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.244459 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:38.248137 (+ 3678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:42235: BlockingRecv error: recv got EOF from 127.0.241.1:42235 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 14:50:38 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:38 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:38.695492 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:38.686884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:41743)
0619 14:50:38.687406 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:38.687449 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:38.687559 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:38.687856 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:38.687859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:38.688283 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:38.688559 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:38.688579 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.689889 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.689893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:38.690961 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:38.690972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.691704 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.691708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:38.691896 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:38.692476 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:38.692493 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:38.695225 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":290,"threads_started":1}
W20260619 14:50:38.695657 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:38.686935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:41743 (local address 127.0.241.62:39039)
0619 14:50:38.687128 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:38.687132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:38.687521 (+ 389us) server_negotiation.cc:407] Connection header received
0619 14:50:38.687648 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:38.687651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:38.687708 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:38.687793 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:38.688707 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.689739 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.691074 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:38.691560 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:38.691653 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:38.695398 (+ 3745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:41743: BlockingRecv error: recv got EOF from 127.0.241.2:41743 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 14:50:39 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:39 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:39.135187 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:39.126570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:58449)
0619 14:50:39.127089 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:39.127104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:39.127215 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:39.127565 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:39.127568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:39.128137 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:39.128325 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:39.128334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:39.129631 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:39.129637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:39.130629 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:39.130640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:39.131423 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:39.131428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:39.131626 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:39.132188 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:39.132204 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:39.134968 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":278,"threads_started":1}
W20260619 14:50:39.135401 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:39.126756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:58449 (local address 127.0.241.62:39039)
0619 14:50:39.126959 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:39.126964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:39.127186 (+ 222us) server_negotiation.cc:407] Connection header received
0619 14:50:39.127334 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:39.127338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:39.127401 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:39.127509 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:39.128461 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:39.129484 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:39.130860 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:39.131273 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:39.131371 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:39.135167 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:58449: BlockingRecv error: recv got EOF from 127.0.241.3:58449 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 14:50:39 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:39 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:39.256413 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:39.248943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:42937)
0619 14:50:39.249411 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:39.249430 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:39.249541 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:39.249830 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:39.249834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:39.250224 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:39.250370 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:39.250377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:39.251310 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:39.251313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:39.252237 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:39.252247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:39.252855 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:39.252858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:39.252956 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:39.253517 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:39.253534 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:39.256181 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":277,"threads_started":1}
W20260619 14:50:39.256594 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:39.249022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:42937 (local address 127.0.241.62:39039)
0619 14:50:39.249278 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:39.249285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:39.249506 (+ 221us) server_negotiation.cc:407] Connection header received
0619 14:50:39.249646 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:39.249650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:39.249706 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:39.249781 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:39.250479 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:39.251200 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:39.252350 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:39.252727 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:39.252825 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:39.256361 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:42937: BlockingRecv error: recv got EOF from 127.0.241.1:42937 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 19 14:50:39 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:39 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:39.704177 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:39.696471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:42407)
0619 14:50:39.696997 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:39.697013 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:39.697131 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:39.697387 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:39.697391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:39.697856 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:39.697990 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:39.697998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:39.699155 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:39.699159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:39.700062 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:39.700072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:39.700694 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:39.700698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:39.700801 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:39.701338 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:39.701355 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:39.703955 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":315,"threads_started":1}
W20260619 14:50:39.704303 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:39.696507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:42407 (local address 127.0.241.62:39039)
0619 14:50:39.696705 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:39.696709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:39.697090 (+ 381us) server_negotiation.cc:407] Connection header received
0619 14:50:39.697219 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:39.697222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:39.697280 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:39.697353 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:39.698122 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:39.699033 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:39.700176 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:39.700561 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:39.700665 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:39.704124 (+ 3459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:42407: BlockingRecv error: recv got EOF from 127.0.241.2:42407 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 14:50:40 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:40 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:40.148905 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:40.136210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:56283)
0619 14:50:40.136980 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:40.136995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:40.137105 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:40.137389 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:40.137392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:40.137777 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:40.137916 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:40.137924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:40.138863 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:40.138867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:40.139962 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:40.139974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:40.140644 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:40.140648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:40.140756 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:40.145547 (+ 4791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:40.145565 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:40.148558 (+ 2993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":545,"threads_started":1}
W20260619 14:50:40.148981 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:40.136327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:56283 (local address 127.0.241.62:39039)
0619 14:50:40.136506 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:40.136512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:40.137079 (+ 567us) server_negotiation.cc:407] Connection header received
0619 14:50:40.137206 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:40.137210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:40.137269 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:40.137347 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:40.138028 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:40.138748 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:40.140108 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:40.140513 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:40.140612 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:40.148748 (+ 8136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:56283: BlockingRecv error: recv got EOF from 127.0.241.3:56283 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 14:50:40 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:40 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:40.265196 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:40.257360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:38441)
0619 14:50:40.257792 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:40.257811 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:40.257922 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:40.258205 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:40.258208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:40.258518 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:40.258688 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:40.258696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:40.259738 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:40.259743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:40.260686 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:40.260699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:40.261334 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:40.261338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:40.261459 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:40.262057 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:40.262074 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:40.264843 (+ 2769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":238,"threads_started":1}
W20260619 14:50:40.265311 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:40.257420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:38441 (local address 127.0.241.62:39039)
0619 14:50:40.257633 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:40.257640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:40.257894 (+ 254us) server_negotiation.cc:407] Connection header received
0619 14:50:40.258002 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:40.258005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:40.258076 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:40.258166 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:40.258826 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:40.259625 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:40.260819 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:40.261209 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:40.261295 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:40.265073 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:38441: BlockingRecv error: recv got EOF from 127.0.241.1:38441 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 14:50:40 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:40 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:40.713452 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:40.705197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:46621)
0619 14:50:40.705746 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:40.705762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:40.705878 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:40.706173 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:40.706176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:40.706492 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:40.706653 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:40.706662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:40.707973 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:40.707980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:40.709204 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:40.709216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:40.709832 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:40.709836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:40.709949 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:40.710589 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:40.710606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:40.713219 (+ 2613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":313,"threads_started":1}
W20260619 14:50:40.713596 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:40.705238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:46621 (local address 127.0.241.62:39039)
0619 14:50:40.705389 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:40.705394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:40.705837 (+ 443us) server_negotiation.cc:407] Connection header received
0619 14:50:40.705980 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:40.705984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:40.706050 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:40.706131 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:40.706785 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:40.707777 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:40.709326 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:40.709717 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:40.709808 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:40.713387 (+ 3579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:46621: BlockingRecv error: recv got EOF from 127.0.241.2:46621 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 14:50:41 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:41 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:41.158123 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:41.149721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:55263)
0619 14:50:41.150232 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:41.150248 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:41.150358 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:41.150691 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:41.150694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:41.151017 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:41.151186 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:41.151194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:41.152210 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:41.152215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:41.153180 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:41.153193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:41.153843 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:41.153847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:41.153983 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:41.154584 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:41.154602 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:41.157726 (+ 3124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":308,"threads_started":1}
W20260619 14:50:41.158209 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:41.149813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:55263 (local address 127.0.241.62:39039)
0619 14:50:41.149969 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:41.149974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:41.150318 (+ 344us) server_negotiation.cc:407] Connection header received
0619 14:50:41.150450 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:41.150453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:41.150530 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:41.150666 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:41.151304 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:41.152054 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:41.153309 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:41.153693 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:41.153778 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:41.157935 (+ 4157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:55263: BlockingRecv error: recv got EOF from 127.0.241.3:55263 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 14:50:41 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:41 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:41.280233 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:41.266110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:56867)
0619 14:50:41.268406 (+ 2296us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:41.268495 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:41.268725 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:41.269335 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:41.269342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:41.269918 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:41.270147 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:41.270160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:41.273302 (+ 3142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:41.273309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:41.274377 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:41.274389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:41.275328 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:41.275334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:41.275566 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:41.276326 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:41.276361 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:41.279910 (+ 3549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2116,"thread_start_us":1975,"threads_started":1}
W20260619 14:50:41.280385 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:41.267242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:56867 (local address 127.0.241.62:39039)
0619 14:50:41.267508 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:41.267518 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:41.268596 (+ 1078us) server_negotiation.cc:407] Connection header received
0619 14:50:41.268886 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:41.268894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:41.269003 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:41.269088 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:41.271984 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:41.273170 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:41.274559 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:41.275156 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:41.275316 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:41.280113 (+ 4797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:56867: BlockingRecv error: recv got EOF from 127.0.241.1:56867 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 14:50:41 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:41 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:41.722216 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:41.714524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:60567)
0619 14:50:41.715074 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:41.715096 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:41.715260 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:41.715625 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:41.715628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:41.715970 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:41.716110 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:41.716117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:41.717031 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:41.717035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:41.717974 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:41.717984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:41.718608 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:41.718612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:41.718733 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:41.719294 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:41.719310 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:41.721998 (+ 2688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":298,"threads_started":1}
W20260619 14:50:41.722424 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:41.714633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:60567 (local address 127.0.241.62:39039)
0619 14:50:41.714808 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:41.714814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:41.715202 (+ 388us) server_negotiation.cc:407] Connection header received
0619 14:50:41.715390 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:41.715394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:41.715491 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:41.715582 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:41.716225 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:41.716929 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:41.718089 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:41.718477 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:41.718568 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:41.722206 (+ 3638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:60567: BlockingRecv error: recv got EOF from 127.0.241.2:60567 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 14:50:42 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:42 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:42.167385 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:42.159309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:46343)
0619 14:50:42.159841 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:42.159860 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:42.159966 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:42.160411 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:42.160415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:42.160853 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:42.160996 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:42.161004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:42.161961 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:42.161964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:42.163078 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:42.163108 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:42.163835 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:42.163839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:42.163961 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:42.164524 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:42.164540 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:42.167169 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":309,"threads_started":1}
W20260619 14:50:42.167639 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:42.159380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:46343 (local address 127.0.241.62:39039)
0619 14:50:42.159562 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:42.159568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:42.159953 (+ 385us) server_negotiation.cc:407] Connection header received
0619 14:50:42.160106 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:42.160110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:42.160223 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:42.160331 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:42.161120 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:42.161854 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:42.163239 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:42.163720 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:42.163812 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:42.167355 (+ 3543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:46343: BlockingRecv error: recv got EOF from 127.0.241.3:46343 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 19 14:50:42 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:42 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:42.289381 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:42.281169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:43869)
0619 14:50:42.281700 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:42.281720 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:42.281874 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:42.282227 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:42.282232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:42.282693 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:42.282848 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:42.282856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:42.283910 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:42.283917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:42.284836 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:42.284847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:42.285544 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:42.285551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:42.285723 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:42.286261 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:42.286279 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:42.289040 (+ 2761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":258,"threads_started":1}
W20260619 14:50:42.289489 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:42.281255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:43869 (local address 127.0.241.62:39039)
0619 14:50:42.281482 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:42.281490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:42.281817 (+ 327us) server_negotiation.cc:407] Connection header received
0619 14:50:42.282005 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:42.282012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:42.282087 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:42.282193 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:42.282993 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:42.283776 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:42.284993 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:42.285389 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:42.285526 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:42.289247 (+ 3721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:43869: BlockingRecv error: recv got EOF from 127.0.241.1:43869 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 14:50:42 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:42 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:42.731752 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:42.723207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:54893)
0619 14:50:42.723829 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:42.723855 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:42.723999 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:42.724351 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:42.724354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:42.724854 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:42.725083 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:42.725097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:42.726392 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:42.726396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:42.727303 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:42.727314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:42.727938 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:42.727942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:42.728061 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:42.728733 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:42.728750 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:42.731494 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":297,"threads_started":1}
W20260619 14:50:42.731897 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:42.723566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:54893 (local address 127.0.241.62:39039)
0619 14:50:42.723775 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:42.723784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:42.723937 (+ 153us) server_negotiation.cc:407] Connection header received
0619 14:50:42.724130 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:42.724137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:42.724200 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:42.724308 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:42.725249 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:42.726273 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:42.727420 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:42.727827 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:42.727917 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:42.731673 (+ 3756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:54893: BlockingRecv error: recv got EOF from 127.0.241.2:54893 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 14:50:43 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:43 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:43.176257 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:43.168317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:58715)
0619 14:50:43.168802 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:43.168819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:43.168935 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:43.169212 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:43.169215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:43.169604 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:43.169757 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:43.169765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:43.170689 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:43.170695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:43.171707 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:43.171720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:43.172319 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:43.172323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:43.172432 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:43.172974 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:43.172990 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:43.175931 (+ 2941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":251,"threads_started":1}
W20260619 14:50:43.176405 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:43.168353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:58715 (local address 127.0.241.62:39039)
0619 14:50:43.168585 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:43.168591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:43.168909 (+ 318us) server_negotiation.cc:407] Connection header received
0619 14:50:43.169029 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:43.169032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:43.169098 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:43.169174 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:43.169887 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:43.170564 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:43.171827 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:43.172209 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:43.172298 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:43.176154 (+ 3856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:58715: BlockingRecv error: recv got EOF from 127.0.241.3:58715 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 14:50:43 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:43 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:43.299295 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:43.290415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:58281)
0619 14:50:43.291027 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:43.291062 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:43.291213 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:43.291639 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:43.291643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:43.292027 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:43.292183 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:43.292192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:43.293264 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:43.293269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:43.294222 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:43.294235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:43.295146 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:43.295151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:43.295372 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:43.296193 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:43.296211 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:43.299045 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":358,"threads_started":1}
W20260619 14:50:43.299525 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:43.290537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:58281 (local address 127.0.241.62:39039)
0619 14:50:43.290739 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:43.290744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:43.291137 (+ 393us) server_negotiation.cc:407] Connection header received
0619 14:50:43.291405 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:43.291409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:43.291491 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:43.291587 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:43.292346 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:43.293114 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:43.294366 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:43.294993 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:43.295118 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:43.299268 (+ 4150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:58281: BlockingRecv error: recv got EOF from 127.0.241.1:58281 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 14:50:43 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:43 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:43.740468 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:43.732604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:50983)
0619 14:50:43.733056 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:43.733071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:43.733186 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:43.733465 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:43.733468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:43.733831 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:43.733976 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:43.733984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:43.734928 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:43.734931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:43.735885 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:43.735896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:43.736558 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:43.736563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:43.736674 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:43.737227 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:43.737242 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:43.740158 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":258,"threads_started":1}
W20260619 14:50:43.740600 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:43.732696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:50983 (local address 127.0.241.62:39039)
0619 14:50:43.732923 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:43.732929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:43.733154 (+ 225us) server_negotiation.cc:407] Connection header received
0619 14:50:43.733273 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:43.733277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:43.733334 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:43.733407 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:43.734085 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:43.734807 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:43.736002 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:43.736419 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:43.736508 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:43.740352 (+ 3844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:50983: BlockingRecv error: recv got EOF from 127.0.241.2:50983 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 14:50:44 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:44 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:44.185276 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:44.177265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:32953)
0619 14:50:44.177755 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:44.177770 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:44.177921 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:44.178261 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:44.178265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:44.178657 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:44.178804 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:44.178812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:44.179908 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:44.179914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:44.180791 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:44.180802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:44.181539 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:44.181545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:44.181711 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:44.182272 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:44.182289 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:44.185042 (+ 2753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":283,"threads_started":1}
W20260619 14:50:44.185545 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:44.177340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:32953 (local address 127.0.241.62:39039)
0619 14:50:44.177564 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:44.177573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:44.177851 (+ 278us) server_negotiation.cc:407] Connection header received
0619 14:50:44.178068 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:44.178071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:44.178130 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:44.178228 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:44.178941 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:44.179777 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:44.180932 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:44.181359 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:44.181484 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:44.185257 (+ 3773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:32953: BlockingRecv error: recv got EOF from 127.0.241.3:32953 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 14:50:44 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:44 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:44.308267 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:44.300182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:37991)
0619 14:50:44.300753 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:44.300773 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:44.300888 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:44.301175 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:44.301178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:44.301692 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:44.301850 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:44.301858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:44.303008 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:44.303014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:44.303921 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:44.303931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:44.304644 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:44.304650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:44.304834 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:44.305355 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:44.305398 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:44.308026 (+ 2628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":295,"threads_started":1}
W20260619 14:50:44.308406 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:44.300278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:37991 (local address 127.0.241.62:39039)
0619 14:50:44.300468 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:44.300473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:44.300842 (+ 369us) server_negotiation.cc:407] Connection header received
0619 14:50:44.300982 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:44.300985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:44.301041 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:44.301127 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:44.301988 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:44.302881 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:44.304071 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:44.304490 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:44.304630 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:44.308203 (+ 3573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:37991: BlockingRecv error: recv got EOF from 127.0.241.1:37991 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 14:50:44 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:44 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:44.750819 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:44.741485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:36637)
0619 14:50:44.742091 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:44.742110 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:44.742250 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:44.742570 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:44.742574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:44.743040 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:44.743240 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:44.743250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:44.744331 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:44.744336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:44.745836 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:44.745849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:44.746572 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:44.746578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:44.746746 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:44.747360 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:44.747381 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:44.750495 (+ 3114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":271,"threads_started":1}
W20260619 14:50:44.750988 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:44.741560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:36637 (local address 127.0.241.62:39039)
0619 14:50:44.741831 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:44.741840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:44.742195 (+ 355us) server_negotiation.cc:407] Connection header received
0619 14:50:44.742371 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:44.742374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:44.742433 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:44.742510 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:44.743366 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:44.744198 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:44.745976 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:44.746435 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:44.746522 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:44.750750 (+ 4228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:36637: BlockingRecv error: recv got EOF from 127.0.241.2:36637 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Jun 19 14:50:45 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:45 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:45.199270 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:45.186163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:34297)
0619 14:50:45.186798 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:45.186820 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:45.186929 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:45.187308 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:45.187313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:45.187877 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:45.188042 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:45.188051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:45.189085 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:45.189090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:45.190003 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:45.190014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:45.190732 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:45.190738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:45.190902 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:45.195936 (+ 5034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:45.195953 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:45.198868 (+ 2915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":283,"threads_started":1}
W20260619 14:50:45.199347 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:45.186283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:34297 (local address 127.0.241.62:39039)
0619 14:50:45.186474 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:45.186480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:45.186904 (+ 424us) server_negotiation.cc:407] Connection header received
0619 14:50:45.187044 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:45.187052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:45.187139 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:45.187217 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:45.188196 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:45.188940 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:45.190148 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:45.190582 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:45.190719 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:45.199083 (+ 8364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:34297: BlockingRecv error: recv got EOF from 127.0.241.3:34297 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 14:50:45 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:45 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:45.317790 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:45.309224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:53497)
0619 14:50:45.309744 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:45.309762 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:45.309911 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:45.310248 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:45.310255 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:45.310722 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:45.310867 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:45.310876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:45.312083 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:45.312095 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:45.312975 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:45.312985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:45.313724 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:45.313730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:45.313901 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:45.314464 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:45.314481 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:45.317558 (+ 3077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":276,"threads_started":1}
W20260619 14:50:45.317963 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:45.309440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:53497 (local address 127.0.241.62:39039)
0619 14:50:45.309660 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:45.309666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:45.309838 (+ 172us) server_negotiation.cc:407] Connection header received
0619 14:50:45.310035 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:45.310041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:45.310114 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:45.310214 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:45.310999 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:45.311944 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:45.313122 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:45.313523 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:45.313687 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:45.317728 (+ 4041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:53497: BlockingRecv error: recv got EOF from 127.0.241.1:53497 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 14:50:45 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:45 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:45.760690 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:45.752094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:44031)
0619 14:50:45.752634 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:45.752651 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:45.752807 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:45.753149 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:45.753153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:45.753557 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:45.753706 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:45.753714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:45.754904 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:45.754910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:45.756028 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:45.756043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:45.756981 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:45.756987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:45.757156 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:45.757783 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:45.757801 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:45.760463 (+ 2662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":280,"threads_started":1}
W20260619 14:50:45.760983 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:45.752208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:44031 (local address 127.0.241.62:39039)
0619 14:50:45.752444 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:45.752450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:45.752739 (+ 289us) server_negotiation.cc:407] Connection header received
0619 14:50:45.752950 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:45.752954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:45.753010 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:45.753130 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:45.753890 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:45.754763 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:45.756192 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:45.756806 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:45.756946 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:45.760687 (+ 3741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:44031: BlockingRecv error: recv got EOF from 127.0.241.2:44031 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 14:50:46 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:46 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:46.208626 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:46.200375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:36999)
0619 14:50:46.200889 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:46.200905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:46.201055 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:46.201417 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:46.201425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:46.201951 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:46.202189 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:46.202202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:46.203244 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:46.203248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:46.204149 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:46.204160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:46.204906 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:46.204913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:46.205042 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:46.205602 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:46.205619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:46.208396 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":260,"threads_started":1}
W20260619 14:50:46.208784 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:46.200439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:36999 (local address 127.0.241.62:39039)
0619 14:50:46.200611 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:46.200615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:46.200987 (+ 372us) server_negotiation.cc:407] Connection header received
0619 14:50:46.201212 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:46.201216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:46.201274 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:46.201357 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:46.202355 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:46.203117 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:46.204264 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:46.204696 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:46.204805 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:46.208573 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:36999: BlockingRecv error: recv got EOF from 127.0.241.3:36999 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 14:50:46 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:46 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:46.327209 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:46.318823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:46751)
0619 14:50:46.319419 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:46.319470 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:46.319618 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:46.320011 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:46.320016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:46.320563 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:46.320794 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:46.320807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:46.321760 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:46.321763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:46.322672 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:46.322682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:46.323304 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:46.323308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:46.323419 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:46.324064 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:46.324088 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:46.326851 (+ 2763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":325,"threads_started":1}
W20260619 14:50:46.327302 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:46.318880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:46751 (local address 127.0.241.62:39039)
0619 14:50:46.319114 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:46.319123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:46.319568 (+ 445us) server_negotiation.cc:407] Connection header received
0619 14:50:46.319731 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:46.319736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:46.319835 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:46.319941 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:46.320940 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:46.321644 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:46.322788 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:46.323182 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:46.323270 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:46.327034 (+ 3764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:46751: BlockingRecv error: recv got EOF from 127.0.241.1:46751 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 14:50:46 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:46 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:46.770601 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:46.761694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:45131)
0619 14:50:46.762222 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:46.762239 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:46.762367 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:46.762744 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:46.762747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:46.763103 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:46.763293 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:46.763330 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:46.764397 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:46.764405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:46.765400 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:46.765411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:46.766132 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:46.766138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:46.766348 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:46.767054 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:46.767070 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:46.770186 (+ 3116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":286,"threads_started":1}
W20260619 14:50:46.770689 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:46.761787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:45131 (local address 127.0.241.62:39039)
0619 14:50:46.761997 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:46.762006 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:46.762320 (+ 314us) server_negotiation.cc:407] Connection header received
0619 14:50:46.762475 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:46.762483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:46.762576 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:46.762703 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:46.763514 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:46.764264 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:46.765542 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:46.765964 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:46.766116 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:46.770415 (+ 4299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:45131: BlockingRecv error: recv got EOF from 127.0.241.2:45131 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 14:50:47 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:47 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:47.218065 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:47.209596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:47633)
0619 14:50:47.210257 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:47.210273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:47.210434 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:47.210771 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:47.210776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:47.211282 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:47.211539 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:47.211556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:47.212909 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:47.212913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:47.213818 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:47.213829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:47.214451 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:47.214455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:47.214569 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:47.215119 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:47.215136 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:47.217833 (+ 2697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":424,"threads_started":1}
W20260619 14:50:47.218196 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:47.209640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:47633 (local address 127.0.241.62:39039)
0619 14:50:47.209883 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:47.209892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:47.210357 (+ 465us) server_negotiation.cc:407] Connection header received
0619 14:50:47.210560 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:47.210571 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:47.210636 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:47.210740 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:47.211738 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:47.212786 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:47.213945 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:47.214325 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:47.214412 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:47.218004 (+ 3592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:47633: BlockingRecv error: recv got EOF from 127.0.241.3:47633 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 14:50:47 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:47 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:47.336630 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:47.328265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:47003)
0619 14:50:47.328852 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:47.328874 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:47.329029 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:47.329408 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:47.329413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:47.329793 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:47.329929 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:47.329937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:47.331032 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:47.331040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:47.332186 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:47.332199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:47.332896 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:47.332900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:47.333017 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:47.333638 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:47.333666 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:47.336396 (+ 2730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":360,"threads_started":1}
W20260619 14:50:47.336954 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:47.328418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:47003 (local address 127.0.241.62:39039)
0619 14:50:47.328659 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:47.328667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:47.328970 (+ 303us) server_negotiation.cc:407] Connection header received
0619 14:50:47.329209 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:47.329213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:47.329269 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:47.329378 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:47.330086 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:47.330895 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:47.332335 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:47.332771 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:47.332867 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:47.336583 (+ 3716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:47003: BlockingRecv error: recv got EOF from 127.0.241.1:47003 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 14:50:47 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:47 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:47.779681 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:47.771885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:45623)
0619 14:50:47.772421 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:47.772437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:47.772541 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:47.772815 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:47.772818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:47.773181 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:47.773334 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:47.773341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:47.774336 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:47.774339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:47.775239 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:47.775248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:47.775935 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:47.775939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:47.776059 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:47.776706 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:47.776722 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:47.779383 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":319,"threads_started":1}
W20260619 14:50:47.779855 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:47.771982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:45623 (local address 127.0.241.62:39039)
0619 14:50:47.772161 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:47.772166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:47.772509 (+ 343us) server_negotiation.cc:407] Connection header received
0619 14:50:47.772635 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:47.772639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:47.772694 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:47.772776 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:47.773476 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:47.774215 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:47.775352 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:47.775802 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:47.775890 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:47.779593 (+ 3703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:45623: BlockingRecv error: recv got EOF from 127.0.241.2:45623 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 14:50:48 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:48 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:48.227203 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:48.219116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:34777)
0619 14:50:48.219657 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:48.219673 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:48.219770 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:48.220135 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:48.220144 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:48.220509 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:48.220658 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:48.220667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.221775 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.221779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:48.222767 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:48.222779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.223693 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.223700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:48.223839 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:48.224481 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:48.224498 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:48.226981 (+ 2483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":346,"threads_started":1}
W20260619 14:50:48.227398 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:48.219291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:34777 (local address 127.0.241.62:39039)
0619 14:50:48.219502 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:48.219508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:48.219740 (+ 232us) server_negotiation.cc:407] Connection header received
0619 14:50:48.219854 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:48.219857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:48.219912 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:48.219988 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:48.220820 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.221594 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.222907 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.223503 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.223626 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:48.227161 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:34777: BlockingRecv error: recv got EOF from 127.0.241.3:34777 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 14:50:48 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:48 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:48.345512 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:48.337528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:32945)
0619 14:50:48.337963 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:48.337983 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:48.338103 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:48.338398 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:48.338401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:48.338880 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:48.339047 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:48.339055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.340110 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.340114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:48.341015 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:48.341028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.341681 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.341685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:48.341793 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:48.342436 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:48.342459 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:48.345287 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":247,"threads_started":1}
W20260619 14:50:48.345678 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:48.337673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:32945 (local address 127.0.241.62:39039)
0619 14:50:48.337884 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:48.337889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:48.338055 (+ 166us) server_negotiation.cc:407] Connection header received
0619 14:50:48.338209 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:48.338213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:48.338289 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:48.338365 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:48.339175 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.339974 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.341150 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.341556 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.341653 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:48.345450 (+ 3797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:32945: BlockingRecv error: recv got EOF from 127.0.241.1:32945 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 14:50:48 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:48 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:48.788868 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:48.780518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:51519)
0619 14:50:48.781052 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:48.781068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:48.781272 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:48.781613 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:48.781617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:48.782135 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:48.782301 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:48.782309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.783467 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.783473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:48.784385 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:48.784396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.785126 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.785133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:48.785288 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:48.785915 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:48.785932 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:48.788638 (+ 2706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":278,"threads_started":1}
W20260619 14:50:48.789045 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:48.780677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:51519 (local address 127.0.241.62:39039)
0619 14:50:48.780898 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:48.780905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:48.781180 (+ 275us) server_negotiation.cc:407] Connection header received
0619 14:50:48.781406 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:48.781411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:48.781478 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:48.781581 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:48.782438 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.783309 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.784528 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.784943 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.785085 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:48.788809 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:51519: BlockingRecv error: recv got EOF from 127.0.241.2:51519 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 14:50:49 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:49 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:49.236521 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.228154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:45517)
0619 14:50:49.228742 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.228763 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:49.228911 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:49.229383 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:49.229390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:49.229878 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:49.230114 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.230126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.231070 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.231078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.232146 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.232158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.232848 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.232854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.233027 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.233570 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:49.233587 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:49.236301 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":311,"threads_started":1}
W20260619 14:50:49.236685 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:49.228309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:45517 (local address 127.0.241.62:39039)
0619 14:50:49.228532 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.228541 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.228878 (+ 337us) server_negotiation.cc:407] Connection header received
0619 14:50:49.229031 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.229050 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.229137 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.229254 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:49.230231 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.230944 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.232288 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.232698 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.232839 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.236458 (+ 3619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:45517: BlockingRecv error: recv got EOF from 127.0.241.3:45517 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 14:50:49 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:49 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:49.354846 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.346465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:59103)
0619 14:50:49.347050 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.347077 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:49.347252 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:49.347658 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:49.347662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:49.348292 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:49.348466 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.348474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.349545 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.349550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.350462 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.350473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.351156 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.351160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.351281 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.351986 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:49.352004 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:49.354620 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":348,"threads_started":1}
W20260619 14:50:49.355033 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:49.346567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:59103 (local address 127.0.241.62:39039)
0619 14:50:49.346844 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.346851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.347166 (+ 315us) server_negotiation.cc:407] Connection header received
0619 14:50:49.347388 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.347399 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.347520 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.347627 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:49.348593 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.349387 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.350582 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.351020 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.351129 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.354781 (+ 3652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:59103: BlockingRecv error: recv got EOF from 127.0.241.1:59103 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 14:50:49 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:49 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:49.797613 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.789791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:33473)
0619 14:50:49.790295 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.790311 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:49.790426 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:49.790708 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:49.790711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:49.791041 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:49.791211 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.791219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.792261 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.792264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.793144 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.793154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.793767 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.793771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.793890 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.794567 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:49.794585 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:49.797366 (+ 2781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":312,"threads_started":1}
W20260619 14:50:49.797775 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:49.789879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:33473 (local address 127.0.241.62:39039)
0619 14:50:49.790103 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.790110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.790390 (+ 280us) server_negotiation.cc:407] Connection header received
0619 14:50:49.790507 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.790510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.790569 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.790654 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:49.791310 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.792148 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.793257 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.793651 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.793741 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.797508 (+ 3767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:33473: BlockingRecv error: recv got EOF from 127.0.241.2:33473 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 14:50:50 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:50 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:50.254017 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.237327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:35211)
0619 14:50:50.237852 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:50.237866 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:50.238029 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:50.238551 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:50.238555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:50.238907 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:50.239054 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.239061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.240639 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.240647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.241559 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.241570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.242172 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.242176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.242292 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.250573 (+ 8281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:50.250592 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:50.253794 (+ 3202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":274,"threads_started":1}
W20260619 14:50:50.254343 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:50.237488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:35211 (local address 127.0.241.62:39039)
0619 14:50:50.237715 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:50.237723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:50.237960 (+ 237us) server_negotiation.cc:407] Connection header received
0619 14:50:50.238217 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:50.238241 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:50.238335 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:50.238440 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:50.239265 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.240496 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.241673 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.242053 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.242144 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.253998 (+ 11854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:35211: BlockingRecv error: recv got EOF from 127.0.241.3:35211 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 14:50:50 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:50 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:50.363996 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.355745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:50877)
0619 14:50:50.356277 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:50.356297 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:50.356409 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:50.356692 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:50.356695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:50.357057 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:50.357223 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.357231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.358194 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.358199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.359654 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.359674 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.360313 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.360316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.360430 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.360984 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:50.361000 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:50.363745 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":284,"threads_started":1}
W20260619 14:50:50.364176 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:50.355957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:50877 (local address 127.0.241.62:39039)
0619 14:50:50.356155 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:50.356160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:50.356370 (+ 210us) server_negotiation.cc:407] Connection header received
0619 14:50:50.356501 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:50.356504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:50.356564 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:50.356652 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:50.357341 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.358040 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.359786 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.360194 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.360287 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.363928 (+ 3641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:50877: BlockingRecv error: recv got EOF from 127.0.241.1:50877 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 14:50:50 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:50 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:50.806313 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.798467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:42095)
0619 14:50:50.798983 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:50.798999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:50.799101 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:50.799376 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:50.799379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:50.799814 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:50.799962 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.799969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.800919 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.800922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.801840 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.801850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.802464 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.802468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.802578 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.803150 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:50.803165 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:50.806019 (+ 2854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":271,"threads_started":1}
W20260619 14:50:50.806432 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:50.798631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:42095 (local address 127.0.241.62:39039)
0619 14:50:50.798889 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:50.798897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:50.799070 (+ 173us) server_negotiation.cc:407] Connection header received
0619 14:50:50.799197 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:50.799200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:50.799259 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:50.799336 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:50.800077 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.800798 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.801954 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.802354 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.802437 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.806205 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:42095: BlockingRecv error: recv got EOF from 127.0.241.2:42095 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 14:50:51 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:51 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:51.263675 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:51.254930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:59601)
0619 14:50:51.255396 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:51.255412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:51.255602 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:51.256016 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:51.256020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:51.256432 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:51.256608 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:51.256618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.257610 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.257614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:51.258614 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:51.258626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.259330 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.259335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:51.259544 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:51.260226 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:51.260248 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:51.263347 (+ 3099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
W20260619 14:50:51.263808 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:51.255024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:59601 (local address 127.0.241.62:39039)
0619 14:50:51.255272 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:51.255279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:51.255563 (+ 284us) server_negotiation.cc:407] Connection header received
0619 14:50:51.255715 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:51.255718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:51.255779 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:51.255858 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:51.256736 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.257469 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.258748 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.259167 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.259255 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:51.263601 (+ 4346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:59601: BlockingRecv error: recv got EOF from 127.0.241.3:59601 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 19 14:50:51 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:51 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:51.373466 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:51.365029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:42271)
0619 14:50:51.365578 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:51.365597 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:51.365757 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:51.366092 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:51.366097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:51.366475 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:51.366641 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:51.366649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.367731 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.367746 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:51.368843 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:51.368856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.369818 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.369823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:51.369994 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:51.370540 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:51.370556 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:51.373140 (+ 2584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":269,"threads_started":1}
W20260619 14:50:51.373565 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:51.365104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:42271 (local address 127.0.241.62:39039)
0619 14:50:51.365484 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:51.365492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:51.365684 (+ 192us) server_negotiation.cc:407] Connection header received
0619 14:50:51.365881 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:51.365887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:51.365956 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:51.366060 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:51.366789 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.367561 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.368981 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.369673 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.369796 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:51.373316 (+ 3520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:42271: BlockingRecv error: recv got EOF from 127.0.241.1:42271 (error 108)
Metrics: {"server-negotiator.queue_time_us":210}
Jun 19 14:50:51 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:51 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:51.815177 10240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:51.807429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:46941)
0619 14:50:51.808024 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:51.808048 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:51.808181 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:51.808460 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:51.808463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:51.808798 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:51.808965 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:51.808973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.809896 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.809900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:51.810823 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:51.810834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.811552 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.811564 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:51.811699 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:51.812330 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:51.812347 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:51.814952 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":282,"threads_started":1}
W20260619 14:50:51.815352 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:51.807530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:46941 (local address 127.0.241.62:39039)
0619 14:50:51.807779 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:51.807786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:51.808113 (+ 327us) server_negotiation.cc:407] Connection header received
0619 14:50:51.808283 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:51.808287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:51.808346 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:51.808422 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:51.809076 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.809779 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.810939 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.811365 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.811531 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:51.815116 (+ 3585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:46941: BlockingRecv error: recv got EOF from 127.0.241.2:46941 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 14:50:52 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:52 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.3@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:52.272881 10241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:52.264549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.3:47101)
0619 14:50:52.265059 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:52.265074 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:52.265243 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:52.265584 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:52.265588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:52.265972 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:52.266152 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:52.266160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.267181 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.267193 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:52.268186 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:52.268198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.268903 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.268908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:52.269081 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:52.269675 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:52.269692 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:52.272546 (+ 2854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":302,"threads_started":1}
W20260619 14:50:52.273056 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:52.264643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:47101 (local address 127.0.241.62:39039)
0619 14:50:52.264879 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:52.264887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:52.265169 (+ 282us) server_negotiation.cc:407] Connection header received
0619 14:50:52.265373 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:52.265378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:52.265445 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:52.265551 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:52.266295 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.267022 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.268324 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.268756 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.268892 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:52.272781 (+ 3889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.3:47101: BlockingRecv error: recv got EOF from 127.0.241.3:47101 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 14:50:52 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:52 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.1@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:52.382984 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:52.374740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.1:41901)
0619 14:50:52.375289 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:52.375308 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:52.375500 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:52.375867 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:52.375872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:52.376315 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:52.376501 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:52.376509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.377492 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.377498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:52.378415 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:52.378425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.379232 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.379237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:52.379417 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:52.380087 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:52.380104 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:52.382621 (+ 2517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":280,"threads_started":1}
W20260619 14:50:52.383018 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:52.374895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:41901 (local address 127.0.241.62:39039)
0619 14:50:52.375122 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:52.375144 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:52.375396 (+ 252us) server_negotiation.cc:407] Connection header received
0619 14:50:52.375636 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:52.375648 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:52.375707 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:52.375814 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:52.376654 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.377357 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.378561 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.379057 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.379204 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:52.382807 (+ 3603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.1:41901: BlockingRecv error: recv got EOF from 127.0.241.1:41901 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 14:50:52 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:50:52 dist-test-slave-p3ml krb5kdc[9198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.241.2@KRBTEST.COM for oryx/127.0.241.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:50:52.824569 10243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:52.816225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:39039 (local address 127.0.241.2:50027)
0619 14:50:52.816764 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:52.816781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:52.816887 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:52.817179 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:52.817182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:52.817561 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:52.817740 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:52.817748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.818845 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.818850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:52.819844 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:52.819855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.820599 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.820605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:52.820787 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:52.821554 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:52.821570 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:52.824316 (+ 2746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.241.62:39039: Server oryx/127.0.241.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":315,"threads_started":1}
W20260619 14:50:52.824707 10198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:52.816321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:50027 (local address 127.0.241.62:39039)
0619 14:50:52.816486 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:52.816490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:52.816856 (+ 366us) server_negotiation.cc:407] Connection header received
0619 14:50:52.816983 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:52.816987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:52.817045 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:52.817124 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:52.817859 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.818691 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.819985 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.820440 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.820576 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:52.824492 (+ 3916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.241.2:50027: BlockingRecv error: recv got EOF from 127.0.241.2:50027 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:50:53.102612 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 9785
I20260619 14:50:53.119993 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.103781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.241.1:46243)
0619 14:50:53.105081 (+ 1300us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.105090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.105118 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:50:53.105192 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.105203 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.105542 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.105746 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:53.107805 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.109313 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.111354 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.112167 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.112311 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.115810 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:53.115840 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:53.115859 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:53.115898 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:53.119123 (+ 3225us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.241.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 14:50:53.119577 (+ 454us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39894: Request ticket server kudu/127.0.241.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":1099,"thread_start_us":937,"threads_started":1}
W20260619 14:50:53.120047 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 9785: Not authorized: Client connection negotiation failed: client connection to 127.0.241.1:46243: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.241.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 14:50:53.120107 10245 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39894: Request ticket server kudu/127.0.241.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 14:50:53.120221 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 9785
I20260619 14:50:53.137549 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 9936
I20260619 14:50:53.149151 10246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.138583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.0.241.2:39059)
0619 14:50:53.139194 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.139201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.139230 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:50:53.139286 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.139292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.139584 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.139810 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:53.140987 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.141990 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.143095 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.143751 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.143820 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.146274 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:53.146281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:53.146293 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:53.146325 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:53.148643 (+ 2318us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.241.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 14:50:53.148878 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37852: Request ticket server kudu/127.0.241.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":288,"threads_started":1}
W20260619 14:50:53.149237 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 9936: Not authorized: Client connection negotiation failed: client connection to 127.0.241.2:39059: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.241.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 14:50:53.149334 10246 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37852: Request ticket server kudu/127.0.241.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 14:50:53.149426 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 9936
I20260619 14:50:53.162722 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 10067
I20260619 14:50:53.173796 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.163540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53048 (local address 127.0.241.3:43671)
0619 14:50:53.164054 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.164061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.164084 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:50:53.164137 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.164143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.164352 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.164491 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:53.165697 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.166699 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.167877 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.168391 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.168484 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.170849 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:53.170857 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:53.170868 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:53.170900 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:53.173311 (+ 2411us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.241.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 14:50:53.173530 (+ 219us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53048: Request ticket server kudu/127.0.241.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":256,"threads_started":1}
W20260619 14:50:53.173889 10247 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53048: Request ticket server kudu/127.0.241.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 14:50:53.173902 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 10067: Not authorized: Client connection negotiation failed: client connection to 127.0.241.3:43671: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.241.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 14:50:53.174140 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 10067
I20260619 14:50:53.190616 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 9716
I20260619 14:50:53.191985 9778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:53.303925 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 9716
2026-06-19T14:50:53Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20914 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T14:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:53Z Disabled control of system clock
I20260619 14:50:53.377031 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33235
--webserver_interface=127.0.241.62
--webserver_port=0
--builtin_ntp_servers=127.0.241.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:33235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:50:53.698678 10252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:53.698949 10252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:53.699086 10252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:53.706298 10252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:53.706395 10252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:53.706462 10252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:53.706565 10252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:53.706627 10252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:53.715590 10252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:33235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:53.716769 10252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:53.718513 10252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:53.729923 10257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:53.729964 10258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:53.729883 10260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:53.732735 10252 server_base.cc:1034] running on GCE node
I20260619 14:50:53.733449 10252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:53.734771 10252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:53.735992 10252 hybrid_clock.cc:648] HybridClock initialized: now 1781880653735958 us; error 50 us; skew 500 ppm
I20260619 14:50:53.736330 10252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:53.738425 10252 webserver.cc:460] Webserver started at http://127.0.241.62:34753/ using document root <none> and password file <none>
I20260619 14:50:53.739085 10252 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:53.739226 10252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:53.739653 10252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:53.742803 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "2bb3772cd36f4c96b14683ea1c5c6d2b"
format_stamp: "Formatted at 2026-06-19 14:50:53 on dist-test-slave-p3ml"
server_key: "2310d7b96316328ba42a07cdf592940b"
server_key_iv: "c595d4662a94220cffbc9d74d2e9ad61"
server_key_version: "encryptionkey@0"
I20260619 14:50:53.743665 10252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "2bb3772cd36f4c96b14683ea1c5c6d2b"
format_stamp: "Formatted at 2026-06-19 14:50:53 on dist-test-slave-p3ml"
server_key: "2310d7b96316328ba42a07cdf592940b"
server_key_iv: "c595d4662a94220cffbc9d74d2e9ad61"
server_key_version: "encryptionkey@0"
I20260619 14:50:53.748684 10252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260619 14:50:53.751957 10266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:53.753202 10252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:50:53.753383 10252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "2bb3772cd36f4c96b14683ea1c5c6d2b"
format_stamp: "Formatted at 2026-06-19 14:50:53 on dist-test-slave-p3ml"
server_key: "2310d7b96316328ba42a07cdf592940b"
server_key_iv: "c595d4662a94220cffbc9d74d2e9ad61"
server_key_version: "encryptionkey@0"
I20260619 14:50:53.753590 10252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:53.787729 10252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:53.788585 10252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:53.788863 10252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:53.806813 10252 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:33235
I20260619 14:50:53.806819 10317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:33235 every 8 connection(s)
I20260619 14:50:53.808233 10252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:53.811844 10318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:53.816181 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 10252
I20260619 14:50:53.816388 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:53.817001 964 external_mini_cluster.cc:1428] Setting key 093afd93493c18a18e002de7dfb8be21
I20260619 14:50:53.822435 10318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b: Bootstrap starting.
I20260619 14:50:53.825448 10318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:53.826774 10318 log.cc:824] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:53.830113 10318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b: No bootstrap required, opened a new log
I20260619 14:50:53.830803 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.818927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.0.241.62:33235)
0619 14:50:53.819891 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.819904 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.819949 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:50:53.821575 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.821595 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.822008 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.822417 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:53.822976 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.824332 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.825698 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.826754 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.826946 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.828116 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:53.828123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:53.828136 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:53.828834 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:53.828930 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:53.828955 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:53.829077 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:53.829283 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 14:50:53.829582 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":306,"threads_started":1}
I20260619 14:50:53.837967 10318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb3772cd36f4c96b14683ea1c5c6d2b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33235 } }
I20260619 14:50:53.838268 10318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:53.838338 10318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb3772cd36f4c96b14683ea1c5c6d2b, State: Initialized, Role: FOLLOWER
I20260619 14:50:53.838843 10318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb3772cd36f4c96b14683ea1c5c6d2b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33235 } }
I20260619 14:50:53.839028 10318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:53.839116 10318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:53.839238 10318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:53.841264 10318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb3772cd36f4c96b14683ea1c5c6d2b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33235 } }
I20260619 14:50:53.841672 10318 leader_election.cc:302] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bb3772cd36f4c96b14683ea1c5c6d2b; no voters:
I20260619 14:50:53.842293 10318 leader_election.cc:288] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:53.842451 10323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:53.843245 10323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [term 1 LEADER]: Becoming Leader. State: Replica: 2bb3772cd36f4c96b14683ea1c5c6d2b, State: Running, Role: LEADER
I20260619 14:50:53.843843 10323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb3772cd36f4c96b14683ea1c5c6d2b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33235 } }
I20260619 14:50:53.844400 10318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:53.846540 10325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bb3772cd36f4c96b14683ea1c5c6d2b. Latest consensus state: current_term: 1 leader_uuid: "2bb3772cd36f4c96b14683ea1c5c6d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb3772cd36f4c96b14683ea1c5c6d2b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33235 } } }
I20260619 14:50:53.846891 10325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:53.846623 10324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bb3772cd36f4c96b14683ea1c5c6d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bb3772cd36f4c96b14683ea1c5c6d2b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33235 } } }
I20260619 14:50:53.847047 10324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:53.847800 10330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:53.854256 10330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:53.864151 10330 catalog_manager.cc:1347] Generated new cluster ID: 58631827546a45ba983392d9a6d3bdb0
I20260619 14:50:53.864280 10330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:53.875283 10330 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:53.876523 10330 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:53.887305 10330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bb3772cd36f4c96b14683ea1c5c6d2b: Generated new TSK 0
I20260619 14:50:53.888101 10330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:50:53.893455 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33235
--builtin_ntp_servers=127.0.241.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 14:50:54.215420 10342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:54.215682 10342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:54.215759 10342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:54.222784 10342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:54.222869 10342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:54.222977 10342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:50:54.231535 10342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:54.232676 10342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:54.234395 10342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:54.247112 10348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:54.247179 10350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:54.247279 10347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:54.248104 10342 server_base.cc:1034] running on GCE node
I20260619 14:50:54.248620 10342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:54.249675 10342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:54.250865 10342 hybrid_clock.cc:648] HybridClock initialized: now 1781880654250805 us; error 51 us; skew 500 ppm
I20260619 14:50:54.251230 10342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:54.253448 10342 webserver.cc:460] Webserver started at http://127.0.241.1:35161/ using document root <none> and password file <none>
I20260619 14:50:54.254102 10342 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:54.254238 10342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:54.254537 10342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:54.257439 10342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "086ded442acb47be97e0df5297797373"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-p3ml"
server_key: "3506fa70dce0eafc090f85b907efe2e7"
server_key_iv: "e9baa18033d0f8f9ef64958d54eb14fd"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.258244 10342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "086ded442acb47be97e0df5297797373"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-p3ml"
server_key: "3506fa70dce0eafc090f85b907efe2e7"
server_key_iv: "e9baa18033d0f8f9ef64958d54eb14fd"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.262908 10342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 14:50:54.266261 10356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.267406 10342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:50:54.267629 10342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "086ded442acb47be97e0df5297797373"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-p3ml"
server_key: "3506fa70dce0eafc090f85b907efe2e7"
server_key_iv: "e9baa18033d0f8f9ef64958d54eb14fd"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.267828 10342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:54.282605 10342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:54.283376 10342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:54.283720 10342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:54.284615 10342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:54.285995 10342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:54.286114 10342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.286247 10342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:54.286332 10342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.318431 10342 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:35679
I20260619 14:50:54.318470 10468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:35679 every 8 connection(s)
I20260619 14:50:54.319819 10342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:54.324688 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 10342
I20260619 14:50:54.324874 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:54.325670 964 external_mini_cluster.cc:1428] Setting key 1f2cd05af6cac0d62325af932dc5c8cd
I20260619 14:50:54.328703 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33235
--builtin_ntp_servers=127.0.241.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:50:54.335063 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.322456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:35525 (local address 127.0.241.62:33235)
0619 14:50:54.322632 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:54.322638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:54.324036 (+ 1398us) server_negotiation.cc:407] Connection header received
0619 14:50:54.326675 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:54.326679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:54.326742 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:54.326862 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:54.328285 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.329147 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.331724 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.332359 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.332483 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.334153 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:54.334162 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:54.334168 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:54.334192 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:54.334228 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:54.334244 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:54.334321 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:54.334585 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 14:50:54.334678 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:50:54.336458 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.322944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33235 (local address 127.0.241.1:35525)
0619 14:50:54.323871 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:54.323910 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:54.326280 (+ 2370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:54.327097 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:54.327104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:54.327211 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:54.328050 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.328075 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.329306 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.329315 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.331569 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.331586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.332593 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.332607 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.332931 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.333765 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:54.333771 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:54.333826 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:54.333927 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:54.334345 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:54.334358 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:54.334369 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 14:50:54.334549 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 14:50:54.334966 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":285,"threads_started":1}
I20260619 14:50:54.337950 10469 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33235
I20260619 14:50:54.338256 10469 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:54.339144 10469 heartbeater.cc:511] Master 127.0.241.62:33235 requested a full tablet report, sending...
I20260619 14:50:54.341225 10283 ts_manager.cc:194] Registered new tserver with Master: 086ded442acb47be97e0df5297797373 (127.0.241.1:35679)
I20260619 14:50:54.342658 10283 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.241.1:35525
W20260619 14:50:54.651966 10472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:54.652235 10472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:54.652369 10472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:54.659742 10472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:54.659849 10472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:54.660009 10472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:50:54.668889 10472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:54.670207 10472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:54.672149 10472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:54.684993 10477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:54.685130 10478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:54.685105 10480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:54.686371 10472 server_base.cc:1034] running on GCE node
I20260619 14:50:54.687012 10472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:54.688076 10472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:54.689337 10472 hybrid_clock.cc:648] HybridClock initialized: now 1781880654689286 us; error 46 us; skew 500 ppm
I20260619 14:50:54.689706 10472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:54.692205 10472 webserver.cc:460] Webserver started at http://127.0.241.2:36215/ using document root <none> and password file <none>
I20260619 14:50:54.693024 10472 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:54.693169 10472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:54.693519 10472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:54.696599 10472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "19e35c200e3e401d9cec635e09a14ae1"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-p3ml"
server_key: "e2ac825bc543cd72e2fab27e1fe6ce6f"
server_key_iv: "b6289bf8ffd9f7757d08acd15f857a8d"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.697433 10472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "19e35c200e3e401d9cec635e09a14ae1"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-p3ml"
server_key: "e2ac825bc543cd72e2fab27e1fe6ce6f"
server_key_iv: "b6289bf8ffd9f7757d08acd15f857a8d"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.702622 10472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:50:54.706033 10486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.707513 10472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:50:54.707707 10472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "19e35c200e3e401d9cec635e09a14ae1"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-p3ml"
server_key: "e2ac825bc543cd72e2fab27e1fe6ce6f"
server_key_iv: "b6289bf8ffd9f7757d08acd15f857a8d"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.707931 10472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:54.720448 10472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:54.721261 10472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:54.721546 10472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:54.722458 10472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:54.723949 10472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:54.724066 10472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.724195 10472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:54.724280 10472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.758991 10472 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:40053
I20260619 14:50:54.759083 10598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:40053 every 8 connection(s)
I20260619 14:50:54.760416 10472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:54.764240 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 10472
I20260619 14:50:54.764428 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:54.765033 964 external_mini_cluster.cc:1428] Setting key c886a871ef69e758c8d0985435cce445
I20260619 14:50:54.767896 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33235
--builtin_ntp_servers=127.0.241.20:43629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:50:54.774123 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.763153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:37043 (local address 127.0.241.62:33235)
0619 14:50:54.763341 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:54.763346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:54.765148 (+ 1802us) server_negotiation.cc:407] Connection header received
0619 14:50:54.767283 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:54.767290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:54.767383 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:54.767523 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:54.768742 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.769528 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.771220 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.771754 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.771902 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.773324 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:54.773331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:54.773334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:54.773352 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:54.773382 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:54.773395 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:54.773492 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:54.773773 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 14:50:54.773847 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 14:50:54.775595 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.763815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33235 (local address 127.0.241.2:37043)
0619 14:50:54.764976 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:54.765012 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:54.767014 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:54.767750 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:54.767758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:54.767864 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:54.768560 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.768579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.769707 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.769712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.771071 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.771084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.771986 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.771996 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.772251 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.773001 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:54.773003 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:54.773051 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:54.773126 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:54.773503 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:54.773515 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:54.773530 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 14:50:54.773754 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 14:50:54.774185 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":354,"threads_started":1}
I20260619 14:50:54.777036 10599 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33235
I20260619 14:50:54.777348 10599 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:54.778136 10599 heartbeater.cc:511] Master 127.0.241.62:33235 requested a full tablet report, sending...
I20260619 14:50:54.779606 10283 ts_manager.cc:194] Registered new tserver with Master: 19e35c200e3e401d9cec635e09a14ae1 (127.0.241.2:40053)
I20260619 14:50:54.780427 10283 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.241.2:37043
W20260619 14:50:55.093845 10602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:55.094208 10602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:55.094383 10602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:55.101589 10602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:55.101745 10602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:55.101925 10602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:50:55.110810 10602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:43629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33235
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:55.112139 10602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:55.113874 10602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:55.126873 10607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:55.126885 10608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:55.127229 10610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:55.133066 10602 server_base.cc:1034] running on GCE node
I20260619 14:50:55.133808 10602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:55.134765 10602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:55.136058 10602 hybrid_clock.cc:648] HybridClock initialized: now 1781880655136011 us; error 67 us; skew 500 ppm
I20260619 14:50:55.136413 10602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:55.138648 10602 webserver.cc:460] Webserver started at http://127.0.241.3:46105/ using document root <none> and password file <none>
I20260619 14:50:55.139424 10602 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:55.139608 10602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:55.139916 10602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:55.142683 10602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "96cab67674a24dacbdf3c6d6fa6e867f"
format_stamp: "Formatted at 2026-06-19 14:50:55 on dist-test-slave-p3ml"
server_key: "29c4348b4715aaeb47bafd2f91004d81"
server_key_iv: "2e8c081beb10c4836088c506b5bf846c"
server_key_version: "encryptionkey@0"
I20260619 14:50:55.143513 10602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "96cab67674a24dacbdf3c6d6fa6e867f"
format_stamp: "Formatted at 2026-06-19 14:50:55 on dist-test-slave-p3ml"
server_key: "29c4348b4715aaeb47bafd2f91004d81"
server_key_iv: "2e8c081beb10c4836088c506b5bf846c"
server_key_version: "encryptionkey@0"
I20260619 14:50:55.148722 10602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 14:50:55.152419 10616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:55.153792 10602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:55.154120 10602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "96cab67674a24dacbdf3c6d6fa6e867f"
format_stamp: "Formatted at 2026-06-19 14:50:55 on dist-test-slave-p3ml"
server_key: "29c4348b4715aaeb47bafd2f91004d81"
server_key_iv: "2e8c081beb10c4836088c506b5bf846c"
server_key_version: "encryptionkey@0"
I20260619 14:50:55.154333 10602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:55.185395 10602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:55.186264 10602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:55.186553 10602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:55.187530 10602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:55.188951 10602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:55.189090 10602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:55.189220 10602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:55.189330 10602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:55.221630 10602 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:41641
I20260619 14:50:55.221715 10728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:41641 every 8 connection(s)
I20260619 14:50:55.222908 10602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:55.230868 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 10602
I20260619 14:50:55.231035 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:55.231606 964 external_mini_cluster.cc:1428] Setting key 03ee1ea16d3f80c16d90d705bb2a67ab
I20260619 14:50:55.235648 10321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:55.225374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:60137 (local address 127.0.241.62:33235)
0619 14:50:55.225608 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:55.225614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:55.227056 (+ 1442us) server_negotiation.cc:407] Connection header received
0619 14:50:55.228771 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:55.228775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:55.228839 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:55.228930 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:55.230056 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.230764 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.233011 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.233498 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.233656 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:55.234962 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:55.234966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:55.234970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:55.234985 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:55.235016 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:55.235027 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:55.235099 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:55.235287 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 14:50:55.235369 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:50:55.236662 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:55.226009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33235 (local address 127.0.241.3:60137)
0619 14:50:55.226884 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:55.226923 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:55.228517 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:55.229122 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:55.229130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:55.229221 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:55.229904 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:55.229921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.230920 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.230927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:55.232875 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:55.232891 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.233716 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.233721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:55.233917 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:55.234674 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:55.234678 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:55.234731 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:55.234804 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:55.235109 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:55.235116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:55.235121 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:50:55.235263 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 14:50:55.235615 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":260,"threads_started":1}
I20260619 14:50:55.237870 10729 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33235
I20260619 14:50:55.238173 10729 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:55.238883 10729 heartbeater.cc:511] Master 127.0.241.62:33235 requested a full tablet report, sending...
I20260619 14:50:55.240177 10283 ts_manager.cc:194] Registered new tserver with Master: 96cab67674a24dacbdf3c6d6fa6e867f (127.0.241.3:41641)
I20260619 14:50:55.240967 10283 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.241.3:60137
I20260619 14:50:55.246546 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:50:55.252703 10321 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:55.251039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.0.241.62:33235)
0619 14:50:55.251184 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:55.251188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:55.251702 (+ 514us) server_negotiation.cc:407] Connection header received
0619 14:50:55.251925 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:55.251929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:55.251986 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:55.252097 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:55.252514 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41422: BlockingRecv error: recv got EOF from 127.0.0.1:41422 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:50:55.253530 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 10342
I20260619 14:50:55.264012 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:55.254416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.0.241.1:35679)
0619 14:50:55.254912 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:55.254919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:55.254945 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:50:55.254996 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:55.255002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:55.255202 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:55.255324 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:55.255722 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.256700 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.257988 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.258618 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.258721 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:55.262673 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:55.262682 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:55.262695 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:55.263244 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:55.263330 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:55.263350 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:55.263488 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:55.263618 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 14:50:55.263729 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":258,"threads_started":1}
I20260619 14:50:55.265226 10464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:55.345850 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 10342
I20260619 14:50:55.362975 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 10472
I20260619 14:50:55.370534 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:55.363941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.0.241.2:40053)
0619 14:50:55.364445 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:55.364452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:55.364478 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:50:55.364533 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:55.364539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:55.364776 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:55.364898 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:55.365281 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.366224 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.367481 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.368073 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.368190 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:55.369137 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:55.369144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:55.369157 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:55.369731 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:55.369772 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:55.369790 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:55.369872 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:55.370048 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 14:50:55.370147 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":249,"threads_started":1}
I20260619 14:50:55.371783 10594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:55.454202 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 10472
I20260619 14:50:55.470522 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 10602
I20260619 14:50:55.478632 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:55.471330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.0.241.3:41641)
0619 14:50:55.472001 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:55.472008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:55.472037 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:50:55.472093 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:55.472099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:55.472319 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:55.472441 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:55.472890 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.473894 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.475357 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.476011 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.476111 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:55.477208 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:55.477217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:55.477230 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:55.477787 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:55.477830 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:55.477848 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:55.477946 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:55.478125 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 14:50:55.478270 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":390,"threads_started":1}
I20260619 14:50:55.479861 10724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:55.563575 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 10602
I20260619 14:50:55.581184 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 10252
I20260619 14:50:55.582464 10313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:55.682760 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 10252
2026-06-19T14:50:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2374 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T14:50:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:55Z Disabled control of system clock
I20260619 14:50:55.753569 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:41687
--webserver_interface=127.0.241.62
--webserver_port=0
--builtin_ntp_servers=127.0.241.20:39135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:41687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:50:56.070652 10748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:56.070943 10748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:56.071026 10748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:56.078235 10748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:56.078346 10748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:56.078382 10748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:56.078433 10748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:56.078478 10748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:56.087806 10748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:39135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:41687
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:41687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:56.089056 10748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:56.090737 10748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:56.102336 10753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:56.102384 10754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:56.102421 10756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:56.103281 10748 server_base.cc:1034] running on GCE node
I20260619 14:50:56.104063 10748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:56.105402 10748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:56.106607 10748 hybrid_clock.cc:648] HybridClock initialized: now 1781880656106564 us; error 39 us; skew 500 ppm
I20260619 14:50:56.106954 10748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:56.109243 10748 webserver.cc:460] Webserver started at http://127.0.241.62:37739/ using document root <none> and password file <none>
I20260619 14:50:56.109915 10748 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:56.110003 10748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:56.110316 10748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:56.113221 10748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "68d0b277e4a641f2a549fd13fb1097e6"
format_stamp: "Formatted at 2026-06-19 14:50:56 on dist-test-slave-p3ml"
server_key: "81c960ede479745842fd3968cb28d919"
server_key_iv: "e2984c29ff1d71a9ab8e170e78e87551"
server_key_version: "encryptionkey@0"
I20260619 14:50:56.113987 10748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "68d0b277e4a641f2a549fd13fb1097e6"
format_stamp: "Formatted at 2026-06-19 14:50:56 on dist-test-slave-p3ml"
server_key: "81c960ede479745842fd3968cb28d919"
server_key_iv: "e2984c29ff1d71a9ab8e170e78e87551"
server_key_version: "encryptionkey@0"
I20260619 14:50:56.118943 10748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 14:50:56.122342 10762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:56.123817 10748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:56.123991 10748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "68d0b277e4a641f2a549fd13fb1097e6"
format_stamp: "Formatted at 2026-06-19 14:50:56 on dist-test-slave-p3ml"
server_key: "81c960ede479745842fd3968cb28d919"
server_key_iv: "e2984c29ff1d71a9ab8e170e78e87551"
server_key_version: "encryptionkey@0"
I20260619 14:50:56.124202 10748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:56.129737 10748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:56.130568 10748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:56.130921 10748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:56.149263 10748 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:41687
I20260619 14:50:56.149312 10813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:41687 every 8 connection(s)
I20260619 14:50:56.150609 10748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:56.154101 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 10748
I20260619 14:50:56.154323 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:56.154546 10814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:56.155150 964 external_mini_cluster.cc:1428] Setting key abe34ac7ce535e7268d71342e102f333
I20260619 14:50:56.163964 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:56.157179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.0.241.62:41687)
0619 14:50:56.158318 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:56.158329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:56.158370 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:50:56.160544 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:56.160565 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:56.160974 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:56.161304 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:56.162376 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:56.162382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:56.162394 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:56.162407 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:56.162468 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:56.162483 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:56.162634 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:56.162758 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 14:50:56.163018 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":490,"threads_started":1}
I20260619 14:50:56.167025 10814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6: Bootstrap starting.
I20260619 14:50:56.169762 10814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:56.170688 10814 log.cc:824] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:56.173276 10814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6: No bootstrap required, opened a new log
I20260619 14:50:56.179970 10814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d0b277e4a641f2a549fd13fb1097e6" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 41687 } }
I20260619 14:50:56.180301 10814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:56.180409 10814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d0b277e4a641f2a549fd13fb1097e6, State: Initialized, Role: FOLLOWER
I20260619 14:50:56.180938 10814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d0b277e4a641f2a549fd13fb1097e6" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 41687 } }
I20260619 14:50:56.181151 10814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:56.181272 10814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:56.181412 10814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:56.183570 10814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d0b277e4a641f2a549fd13fb1097e6" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 41687 } }
I20260619 14:50:56.184044 10814 leader_election.cc:302] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68d0b277e4a641f2a549fd13fb1097e6; no voters:
I20260619 14:50:56.184634 10814 leader_election.cc:288] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:56.184787 10819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:56.185446 10819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [term 1 LEADER]: Becoming Leader. State: Replica: 68d0b277e4a641f2a549fd13fb1097e6, State: Running, Role: LEADER
I20260619 14:50:56.185992 10819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d0b277e4a641f2a549fd13fb1097e6" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 41687 } }
I20260619 14:50:56.186551 10814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:56.188570 10821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68d0b277e4a641f2a549fd13fb1097e6. Latest consensus state: current_term: 1 leader_uuid: "68d0b277e4a641f2a549fd13fb1097e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d0b277e4a641f2a549fd13fb1097e6" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 41687 } } }
I20260619 14:50:56.188665 10820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68d0b277e4a641f2a549fd13fb1097e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d0b277e4a641f2a549fd13fb1097e6" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 41687 } } }
I20260619 14:50:56.189033 10821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:56.189028 10820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:56.189929 10824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:56.195335 10824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:56.208678 10824 catalog_manager.cc:1347] Generated new cluster ID: b1362b3ce52c4b72848f90af649670ee
I20260619 14:50:56.208817 10824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:56.221942 10824 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:56.222412 10824 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:56.232476 10824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68d0b277e4a641f2a549fd13fb1097e6: Generated new TSK 0
I20260619 14:50:56.233477 10824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:50:56.250403 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:41687
--builtin_ntp_servers=127.0.241.20:39135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:50:56.595276 10838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:56.595623 10838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:56.595806 10838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:56.603088 10838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:56.603193 10838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:56.603348 10838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:50:56.612736 10838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:39135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:41687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:56.614022 10838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:56.615919 10838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:56.629105 10843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:56.629057 10846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:56.629135 10844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:56.630014 10838 server_base.cc:1034] running on GCE node
I20260619 14:50:56.630652 10838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:56.631690 10838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:56.632912 10838 hybrid_clock.cc:648] HybridClock initialized: now 1781880656632857 us; error 50 us; skew 500 ppm
I20260619 14:50:56.633216 10838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:56.635712 10838 webserver.cc:460] Webserver started at http://127.0.241.1:46855/ using document root <none> and password file <none>
I20260619 14:50:56.636364 10838 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:56.636492 10838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:56.636801 10838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:56.639624 10838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "08d9a5fc93464ddba2d8d7abaf43bb5f"
format_stamp: "Formatted at 2026-06-19 14:50:56 on dist-test-slave-p3ml"
server_key: "13214cc5ec9c50cf9f2ce4e9d6f4464a"
server_key_iv: "57fdcb12ef216cde2d4b2e47be444e3f"
server_key_version: "encryptionkey@0"
I20260619 14:50:56.640383 10838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "08d9a5fc93464ddba2d8d7abaf43bb5f"
format_stamp: "Formatted at 2026-06-19 14:50:56 on dist-test-slave-p3ml"
server_key: "13214cc5ec9c50cf9f2ce4e9d6f4464a"
server_key_iv: "57fdcb12ef216cde2d4b2e47be444e3f"
server_key_version: "encryptionkey@0"
I20260619 14:50:56.645232 10838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 14:50:56.648483 10852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:56.649688 10838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:50:56.649868 10838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "08d9a5fc93464ddba2d8d7abaf43bb5f"
format_stamp: "Formatted at 2026-06-19 14:50:56 on dist-test-slave-p3ml"
server_key: "13214cc5ec9c50cf9f2ce4e9d6f4464a"
server_key_iv: "57fdcb12ef216cde2d4b2e47be444e3f"
server_key_version: "encryptionkey@0"
I20260619 14:50:56.650092 10838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:56.655599 10838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:56.656333 10838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:56.656605 10838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:56.657457 10838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:56.658751 10838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:56.658870 10838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:56.659005 10838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:56.659097 10838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:56.691277 10838 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:44717
I20260619 14:50:56.691324 10964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:44717 every 8 connection(s)
I20260619 14:50:56.692755 10838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:56.696416 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 10838
I20260619 14:50:56.696626 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:56.697304 964 external_mini_cluster.cc:1428] Setting key 390b66efc6b67ae5b506cec3fcde6c60
I20260619 14:50:56.699992 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:41687
--builtin_ntp_servers=127.0.241.20:39135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 14:50:56.701311 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:56.695180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:56473 (local address 127.0.241.62:41687)
0619 14:50:56.695620 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:56.695624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:56.697064 (+ 1440us) server_negotiation.cc:407] Connection header received
0619 14:50:56.698906 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:56.698911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:56.698978 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:56.699071 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:56.700476 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:56.700484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:56.700491 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:56.700515 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:56.700568 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:56.700587 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:56.700714 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:56.700876 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 14:50:56.700938 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":202,"threads_started":1}
I20260619 14:50:56.702005 10968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:56.695799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:41687 (local address 127.0.241.1:56473)
0619 14:50:56.696892 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:56.696936 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:56.698667 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:56.699308 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:56.699316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:56.699402 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:56.700154 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:56.700157 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:56.700209 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:56.700312 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:56.700689 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:56.700696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:56.700702 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:50:56.700902 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 14:50:56.701082 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":346,"threads_started":1}
I20260619 14:50:56.703488 10965 heartbeater.cc:347] Connected to a master server at 127.0.241.62:41687
I20260619 14:50:56.703774 10965 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:56.704301 10965 heartbeater.cc:511] Master 127.0.241.62:41687 requested a full tablet report, sending...
I20260619 14:50:56.706279 10779 ts_manager.cc:194] Registered new tserver with Master: 08d9a5fc93464ddba2d8d7abaf43bb5f (127.0.241.1:44717)
W20260619 14:50:57.017553 10969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:57.017804 10969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:57.017956 10969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:57.025275 10969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:57.025364 10969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:57.025534 10969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:50:57.034582 10969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:39135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:41687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:57.035921 10969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:57.037622 10969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:57.050137 10975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:57.050179 10977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:57.050829 10974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:57.051376 10969 server_base.cc:1034] running on GCE node
I20260619 14:50:57.051962 10969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:57.052891 10969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:57.054083 10969 hybrid_clock.cc:648] HybridClock initialized: now 1781880657054029 us; error 52 us; skew 500 ppm
I20260619 14:50:57.054387 10969 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:57.056488 10969 webserver.cc:460] Webserver started at http://127.0.241.2:37615/ using document root <none> and password file <none>
I20260619 14:50:57.057108 10969 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:57.057188 10969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:57.057451 10969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:57.060206 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "9da709ce39004e06a78fa01a8ec0f276"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-p3ml"
server_key: "6426cfdc1c0a415bb4dff225d7490253"
server_key_iv: "8f850cda88c70f66b477fc7dd14dc8cc"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.060904 10969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "9da709ce39004e06a78fa01a8ec0f276"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-p3ml"
server_key: "6426cfdc1c0a415bb4dff225d7490253"
server_key_iv: "8f850cda88c70f66b477fc7dd14dc8cc"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.066148 10969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:50:57.069362 10983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.070614 10969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:50:57.070739 10969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "9da709ce39004e06a78fa01a8ec0f276"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-p3ml"
server_key: "6426cfdc1c0a415bb4dff225d7490253"
server_key_iv: "8f850cda88c70f66b477fc7dd14dc8cc"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.070919 10969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:57.076298 10969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:57.076977 10969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:57.077194 10969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:57.077956 10969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:57.079182 10969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:57.079263 10969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.079339 10969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:57.079393 10969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.111891 10969 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:38077
I20260619 14:50:57.111976 11095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:38077 every 8 connection(s)
I20260619 14:50:57.113181 10969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:57.120654 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.115600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:52501 (local address 127.0.241.62:41687)
0619 14:50:57.115796 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.115800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.117068 (+ 1268us) server_negotiation.cc:407] Connection header received
0619 14:50:57.118593 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.118597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.118646 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.118733 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:57.120015 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:57.120018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:57.120021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:57.120033 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:57.120065 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:57.120075 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:57.120162 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.120338 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.120407 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:50:57.121590 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.116094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:41687 (local address 127.0.241.2:52501)
0619 14:50:57.116915 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:57.116950 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:57.118373 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:57.118921 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:57.118928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:57.119010 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:57.119769 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:57.119771 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:57.119805 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:57.119884 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:57.120166 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:57.120172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:57.120178 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:50:57.120385 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 14:50:57.120563 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":245,"threads_started":1}
I20260619 14:50:57.121699 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 10969
I20260619 14:50:57.121891 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:57.122426 964 external_mini_cluster.cc:1428] Setting key 4e0ce5f636206b719ef5d80ffd632879
I20260619 14:50:57.122982 11096 heartbeater.cc:347] Connected to a master server at 127.0.241.62:41687
I20260619 14:50:57.123252 11096 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:57.123980 11096 heartbeater.cc:511] Master 127.0.241.62:41687 requested a full tablet report, sending...
I20260619 14:50:57.125392 10779 ts_manager.cc:194] Registered new tserver with Master: 9da709ce39004e06a78fa01a8ec0f276 (127.0.241.2:38077)
I20260619 14:50:57.125424 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:41687
--builtin_ntp_servers=127.0.241.20:39135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:50:57.446521 11099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:57.446784 11099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:57.446869 11099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:57.454099 11099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:57.454219 11099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:57.454350 11099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:50:57.463114 11099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:39135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:41687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:57.464341 11099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:57.466221 11099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:57.479156 11105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:57.479283 11104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:57.479564 11107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:57.480674 11099 server_base.cc:1034] running on GCE node
I20260619 14:50:57.481333 11099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:57.482324 11099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:57.483557 11099 hybrid_clock.cc:648] HybridClock initialized: now 1781880657483487 us; error 44 us; skew 500 ppm
I20260619 14:50:57.483889 11099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:50:57.486266 11099 webserver.cc:460] Webserver started at http://127.0.241.3:41425/ using document root <none> and password file <none>
I20260619 14:50:57.487061 11099 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:57.487195 11099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:57.487664 11099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:57.490592 11099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "57f5e61922ec449aaf846f89c5b89727"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-p3ml"
server_key: "ff0aa3b79643e2177a0625ecd849e413"
server_key_iv: "08e589438c39293f400d395d87221bb0"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.491487 11099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "57f5e61922ec449aaf846f89c5b89727"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-p3ml"
server_key: "ff0aa3b79643e2177a0625ecd849e413"
server_key_iv: "08e589438c39293f400d395d87221bb0"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.496693 11099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:50:57.500247 11113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.501559 11099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:57.501683 11099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "57f5e61922ec449aaf846f89c5b89727"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-p3ml"
server_key: "ff0aa3b79643e2177a0625ecd849e413"
server_key_iv: "08e589438c39293f400d395d87221bb0"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.501863 11099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:57.507359 11099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:57.508155 11099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:57.508384 11099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:57.509197 11099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:57.510471 11099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:57.510572 11099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.510649 11099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:57.510702 11099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.542663 11099 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:41473
I20260619 14:50:57.542754 11225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:41473 every 8 connection(s)
I20260619 14:50:57.544008 11099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:57.547164 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 11099
I20260619 14:50:57.547348 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:57.547911 964 external_mini_cluster.cc:1428] Setting key d520899dbc69c83d502c0fc6f263ce39
I20260619 14:50:57.552536 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.546492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:53295 (local address 127.0.241.62:41687)
0619 14:50:57.546677 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.546681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.548323 (+ 1642us) server_negotiation.cc:407] Connection header received
0619 14:50:57.550291 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.550296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.550358 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.550491 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:57.551836 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:57.551841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:57.551846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:57.551862 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:57.551899 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:57.551910 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:57.552041 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.552224 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.552285 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:50:57.553504 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.547124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:41687 (local address 127.0.241.3:53295)
0619 14:50:57.548168 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:57.548210 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:57.550027 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:57.550759 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:57.550771 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:57.550898 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:57.551535 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:57.551537 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:57.551570 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:57.551648 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:57.552010 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:57.552022 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:57.552031 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:50:57.552299 (+ 268us) client_negotiation.cc:240] Negotiation successful
0619 14:50:57.552544 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":340,"threads_started":1}
I20260619 14:50:57.554803 11226 heartbeater.cc:347] Connected to a master server at 127.0.241.62:41687
I20260619 14:50:57.555097 11226 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:57.555718 11226 heartbeater.cc:511] Master 127.0.241.62:41687 requested a full tablet report, sending...
I20260619 14:50:57.557022 10779 ts_manager.cc:194] Registered new tserver with Master: 57f5e61922ec449aaf846f89c5b89727 (127.0.241.3:41473)
I20260619 14:50:57.562894 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:50:57.569375 10967 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:57.567518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.0.241.62:41687)
0619 14:50:57.567741 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.567749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.568300 (+ 551us) server_negotiation.cc:407] Connection header received
0619 14:50:57.568502 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.568506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.568575 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.568681 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:57.569132 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60746: BlockingRecv error: recv got EOF from 127.0.0.1:60746 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:50:57.569896 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 10838
I20260619 14:50:57.574712 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.571295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.0.241.1:44717)
0619 14:50:57.572375 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.572384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.572413 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:50:57.572504 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.572514 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.572781 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.572948 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:57.573883 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:57.573893 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:57.573907 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:57.573953 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:57.574010 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:57.574032 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:57.574168 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.574303 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.574398 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":791,"threads_started":1}
I20260619 14:50:57.575925 10960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:57.655956 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 10838
I20260619 14:50:57.672705 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 10969
I20260619 14:50:57.676357 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.673700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.0.241.2:38077)
0619 14:50:57.674229 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.674239 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.674276 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:50:57.674338 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.674344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.674568 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.674693 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:57.675654 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:57.675661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:57.675673 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:57.675690 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:57.675732 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:57.675746 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:57.675859 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.675936 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.676059 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":234,"threads_started":1}
I20260619 14:50:57.677520 11091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:57.758281 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 10969
I20260619 14:50:57.774216 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 11099
I20260619 14:50:57.778292 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.775229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54974 (local address 127.0.241.3:41473)
0619 14:50:57.775894 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.775901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.775929 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:50:57.775989 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.775997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.776216 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.776389 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:57.777545 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:57.777551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:57.777563 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:57.777579 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:57.777617 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:57.777630 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:57.777816 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.777888 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.777987 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":373,"threads_started":1}
I20260619 14:50:57.779618 11221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:57.859556 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 11099
I20260619 14:50:57.875267 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 10748
I20260619 14:50:57.876600 10809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:50:57.975946 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 10748
2026-06-19T14:50:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2292 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:50:58 dist-test-slave-p3ml krb5kdc[11244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:50:58 dist-test-slave-p3ml krb5kdc[11244](info): set up 2 sockets
Jun 19 14:50:58 dist-test-slave-p3ml krb5kdc[11244](info): commencing operation
krb5kdc: starting...
W20260619 14:50:59.080534 964 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:50:59 dist-test-slave-p3ml krb5kdc[11244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:50:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:50:59.307073 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:38563
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:36507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:38563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:59.627242 11257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:59.627533 11257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:59.627713 11257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:59.634784 11257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:59.634884 11257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:59.634985 11257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:59.635066 11257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:59.635134 11257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:59.644891 11257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:36507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:38563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:38563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:50:59.646209 11257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:59.647522 11257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:59.658759 11262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:59.658715 11265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:59.658735 11263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:59.659570 11257 server_base.cc:1034] running on GCE node
I20260619 14:50:59.660261 11257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:59.661609 11257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:59.662817 11257 hybrid_clock.cc:648] HybridClock initialized: now 1781880659662783 us; error 29 us; skew 500 ppm
I20260619 14:50:59.663156 11257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:50:59 dist-test-slave-p3ml krb5kdc[11244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:59.666232 11257 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:50:59.667677 11257 webserver.cc:460] Webserver started at http://127.0.241.62:35765/ using document root <none> and password file <none>
I20260619 14:50:59.668311 11257 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:59.668406 11257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:59.668653 11257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:59.671494 11257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "a08f5b33ff9e495e91a8b357b01f1d0d"
format_stamp: "Formatted at 2026-06-19 14:50:59 on dist-test-slave-p3ml"
server_key: "d89278afaea6eafa385f1d52f2b6920d"
server_key_iv: "020f144ab867db38b510ad9686c7ad01"
server_key_version: "encryptionkey@0"
I20260619 14:50:59.672223 11257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "a08f5b33ff9e495e91a8b357b01f1d0d"
format_stamp: "Formatted at 2026-06-19 14:50:59 on dist-test-slave-p3ml"
server_key: "d89278afaea6eafa385f1d52f2b6920d"
server_key_iv: "020f144ab867db38b510ad9686c7ad01"
server_key_version: "encryptionkey@0"
I20260619 14:50:59.677139 11257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:50:59.680342 11272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:59.681644 11257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:50:59.681829 11257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "a08f5b33ff9e495e91a8b357b01f1d0d"
format_stamp: "Formatted at 2026-06-19 14:50:59 on dist-test-slave-p3ml"
server_key: "d89278afaea6eafa385f1d52f2b6920d"
server_key_iv: "020f144ab867db38b510ad9686c7ad01"
server_key_version: "encryptionkey@0"
I20260619 14:50:59.682016 11257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:59.698834 11257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:59.702605 11257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:59.702907 11257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:59.721408 11257 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:38563
I20260619 14:50:59.721439 11323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:38563 every 8 connection(s)
I20260619 14:50:59.722721 11257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:59.726379 11324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:59.727855 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 11257
I20260619 14:50:59.728135 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:50:59.728746 964 external_mini_cluster.cc:1428] Setting key f2b85285848cc0d012753778d89cb827
I20260619 14:50:59.738044 11324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d: Bootstrap starting.
I20260619 14:50:59.741809 11324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:59.743043 11324 log.cc:824] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:50:59 dist-test-slave-p3ml krb5kdc[11244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:50:59.745803 11324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d: No bootstrap required, opened a new log
I20260619 14:50:59.752286 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:59.730728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53222 (local address 127.0.241.62:38563)
0619 14:50:59.731553 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:59.731564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:59.731609 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:50:59.733256 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:59.733277 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:59.733658 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:59.733956 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:59.735724 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:59.736943 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:59.738725 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:59.739354 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:59.739545 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:59.744270 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:59.744278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:59.744293 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:59.744329 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:59.747961 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:59.748638 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:59.748646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:59.748653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:59.748863 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:59.749204 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:59.749210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:59.749213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:59.749918 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:59.750238 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:59.750526 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 14:50:59.750942 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":364,"threads_started":1}
I20260619 14:50:59.755942 11324 raft_consensus.cc:348] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08f5b33ff9e495e91a8b357b01f1d0d" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38563 } }
I20260619 14:50:59.756317 11324 raft_consensus.cc:374] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:59.756423 11324 raft_consensus.cc:729] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a08f5b33ff9e495e91a8b357b01f1d0d, State: Initialized, Role: FOLLOWER
I20260619 14:50:59.756950 11324 consensus_queue.cc:260] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08f5b33ff9e495e91a8b357b01f1d0d" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38563 } }
I20260619 14:50:59.757192 11324 raft_consensus.cc:388] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:59.757331 11324 raft_consensus.cc:482] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:59.757505 11324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:59.760026 11324 raft_consensus.cc:504] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08f5b33ff9e495e91a8b357b01f1d0d" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38563 } }
I20260619 14:50:59.760483 11324 leader_election.cc:302] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a08f5b33ff9e495e91a8b357b01f1d0d; no voters:
I20260619 14:50:59.761066 11324 leader_election.cc:288] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:59.761197 11329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:59.761890 11329 raft_consensus.cc:686] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [term 1 LEADER]: Becoming Leader. State: Replica: a08f5b33ff9e495e91a8b357b01f1d0d, State: Running, Role: LEADER
I20260619 14:50:59.762369 11329 consensus_queue.cc:237] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08f5b33ff9e495e91a8b357b01f1d0d" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38563 } }
I20260619 14:50:59.763113 11324 sys_catalog.cc:564] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:59.765311 11331 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a08f5b33ff9e495e91a8b357b01f1d0d. Latest consensus state: current_term: 1 leader_uuid: "a08f5b33ff9e495e91a8b357b01f1d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08f5b33ff9e495e91a8b357b01f1d0d" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38563 } } }
I20260619 14:50:59.765708 11331 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:59.766708 11330 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a08f5b33ff9e495e91a8b357b01f1d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08f5b33ff9e495e91a8b357b01f1d0d" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38563 } } }
I20260619 14:50:59.767125 11330 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:59.768291 11338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:59.772723 11338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:59.781046 11338 catalog_manager.cc:1347] Generated new cluster ID: ac0f6eafcb594f72b46972bfcfcd422e
I20260619 14:50:59.781164 11338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:59.799172 11338 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:59.800515 11338 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:59.810551 11338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d: Generated new TSK 0
I20260619 14:50:59.811390 11338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:50:59.866691 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:38563
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:36507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:00.196228 11350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:00.196506 11350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:00.196599 11350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:00.203672 11350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:00.203781 11350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:00.203895 11350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:51:00.213445 11350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:36507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:38563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:51:00.214632 11350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:00.215873 11350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:00.231746 11355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:00.233903 11356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:00.233959 11358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:00.234543 11350 server_base.cc:1034] running on GCE node
I20260619 14:51:00.235240 11350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:00.236196 11350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:00.237425 11350 hybrid_clock.cc:648] HybridClock initialized: now 1781880660237385 us; error 53 us; skew 500 ppm
I20260619 14:51:00.237733 11350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:51:00 dist-test-slave-p3ml krb5kdc[11244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:00.240809 11350 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:51:00.242172 11350 webserver.cc:460] Webserver started at http://127.0.241.1:44849/ using document root <none> and password file <none>
I20260619 14:51:00.242874 11350 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:00.243019 11350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:00.243391 11350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:00.246402 11350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "e0926e6797794322ac541cb1e747c177"
format_stamp: "Formatted at 2026-06-19 14:51:00 on dist-test-slave-p3ml"
server_key: "970ab056c5d1c376d18bf9d338aba2c6"
server_key_iv: "5fd216f510caec0fa015da267b07bac7"
server_key_version: "encryptionkey@0"
I20260619 14:51:00.247141 11350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "e0926e6797794322ac541cb1e747c177"
format_stamp: "Formatted at 2026-06-19 14:51:00 on dist-test-slave-p3ml"
server_key: "970ab056c5d1c376d18bf9d338aba2c6"
server_key_iv: "5fd216f510caec0fa015da267b07bac7"
server_key_version: "encryptionkey@0"
I20260619 14:51:00.252264 11350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 14:51:00.255517 11365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:00.256834 11350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:51:00.257030 11350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "e0926e6797794322ac541cb1e747c177"
format_stamp: "Formatted at 2026-06-19 14:51:00 on dist-test-slave-p3ml"
server_key: "970ab056c5d1c376d18bf9d338aba2c6"
server_key_iv: "5fd216f510caec0fa015da267b07bac7"
server_key_version: "encryptionkey@0"
I20260619 14:51:00.257222 11350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:00.291584 11350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:00.295950 11350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:00.296245 11350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:00.297169 11350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:00.298439 11350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:00.298547 11350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:00.298689 11350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:00.298769 11350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:00.330843 11350 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:33309
I20260619 14:51:00.330899 11477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:33309 every 8 connection(s)
I20260619 14:51:00.332207 11350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:51:00.339330 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 11350
I20260619 14:51:00.339596 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:51:00.340279 964 external_mini_cluster.cc:1428] Setting key bd209a7ceffbe95cfba1d3f9128188ec
Jun 19 14:51:00 dist-test-slave-p3ml krb5kdc[11244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:51:00.353929 11480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:00.334654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:60653 (local address 127.0.241.62:38563)
0619 14:51:00.335126 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:00.335132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:00.336185 (+ 1053us) server_negotiation.cc:407] Connection header received
0619 14:51:00.337928 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:00.337932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:00.338045 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:00.338158 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:00.339921 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:00.340702 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:00.342484 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:00.343134 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:00.343303 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:00.347043 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:00.347053 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:00.347061 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:00.347110 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:00.350675 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:00.351666 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:00.351670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:00.351673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:00.351801 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:00.352290 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:00.352293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:00.352295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:00.352553 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:00.352721 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:00.353532 (+ 811us) server_negotiation.cc:299] Negotiation successful
0619 14:51:00.353699 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":215,"threads_started":1}
I20260619 14:51:00.355407 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:00.335280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:38563 (local address 127.0.241.1:60653)
0619 14:51:00.336023 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:00.336060 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:00.337654 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:00.338356 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:00.338363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:00.338831 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:00.339740 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:00.339768 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:00.340888 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:00.340894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:00.342310 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:00.342321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:00.343393 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:00.343401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:00.343692 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:00.344403 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:00.344424 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:00.346780 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:00.350824 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:00.350834 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:00.350853 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:00.351520 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:00.351913 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:00.351920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:00.351927 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:00.352167 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:00.352685 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:00.352696 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:00.353129 (+ 433us) client_negotiation.cc:769] Sending connection context
0619 14:51:00.353419 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 14:51:00.353950 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":343,"threads_started":1}
I20260619 14:51:00.356881 11478 heartbeater.cc:347] Connected to a master server at 127.0.241.62:38563
I20260619 14:51:00.357187 11478 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:00.357867 11478 heartbeater.cc:511] Master 127.0.241.62:38563 requested a full tablet report, sending...
I20260619 14:51:00.359926 11289 ts_manager.cc:194] Registered new tserver with Master: e0926e6797794322ac541cb1e747c177 (127.0.241.1:33309)
I20260619 14:51:00.361552 11289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:60653
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:51:00.394290 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:38563
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:36507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:00.741302 11484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:00.741601 11484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:00.741775 11484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:00.749567 11484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:00.749716 11484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:00.749892 11484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:51:00.759347 11484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:36507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:38563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:51:00.760625 11484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:00.761899 11484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:00.774295 11489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:00.774408 11492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:00.774425 11490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:00.775221 11484 server_base.cc:1034] running on GCE node
I20260619 14:51:00.775874 11484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:00.776767 11484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:00.777984 11484 hybrid_clock.cc:648] HybridClock initialized: now 1781880660777948 us; error 50 us; skew 500 ppm
I20260619 14:51:00.778291 11484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:51:00 dist-test-slave-p3ml krb5kdc[11244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:00.781323 11484 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:51:00.782655 11484 webserver.cc:460] Webserver started at http://127.0.241.2:33175/ using document root <none> and password file <none>
I20260619 14:51:00.783322 11484 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:00.783495 11484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:00.783774 11484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:00.786556 11484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "478742ce5ad94cc98f8a3ff1ea814802"
format_stamp: "Formatted at 2026-06-19 14:51:00 on dist-test-slave-p3ml"
server_key: "afbc65dfa3e8d098fd80be35ff73e9ce"
server_key_iv: "42921a53c52feb64903d6f3d2a689cde"
server_key_version: "encryptionkey@0"
I20260619 14:51:00.787313 11484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "478742ce5ad94cc98f8a3ff1ea814802"
format_stamp: "Formatted at 2026-06-19 14:51:00 on dist-test-slave-p3ml"
server_key: "afbc65dfa3e8d098fd80be35ff73e9ce"
server_key_iv: "42921a53c52feb64903d6f3d2a689cde"
server_key_version: "encryptionkey@0"
I20260619 14:51:00.792158 11484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 14:51:00.795383 11499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:00.796661 11484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:51:00.796844 11484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "478742ce5ad94cc98f8a3ff1ea814802"
format_stamp: "Formatted at 2026-06-19 14:51:00 on dist-test-slave-p3ml"
server_key: "afbc65dfa3e8d098fd80be35ff73e9ce"
server_key_iv: "42921a53c52feb64903d6f3d2a689cde"
server_key_version: "encryptionkey@0"
I20260619 14:51:00.797031 11484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:00.812462 11484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:00.816670 11484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:00.816956 11484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:00.817903 11484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:00.819350 11484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:00.819473 11484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 14:51:00.819618 11484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:00.819705 11484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:00.852651 11484 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:42885
I20260619 14:51:00.852699 11611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:42885 every 8 connection(s)
I20260619 14:51:00.853963 11484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:51:00.856520 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 11484
I20260619 14:51:00.856729 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:51:00.857276 964 external_mini_cluster.cc:1428] Setting key 85964ff589c2fab2d7aa941fd559c3e4
Jun 19 14:51:00 dist-test-slave-p3ml krb5kdc[11244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:51:00.874166 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:00.856696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:52253 (local address 127.0.241.62:38563)
0619 14:51:00.857176 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:00.857179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:00.858339 (+ 1160us) server_negotiation.cc:407] Connection header received
0619 14:51:00.860345 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:00.860350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:00.860440 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:00.860557 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:00.862057 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:00.862820 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:00.864456 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:00.864872 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:00.865011 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:00.868808 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:00.868813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:00.868816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:00.868845 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:00.871266 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:00.871983 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:00.871989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:00.872000 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:00.872111 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:00.872536 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:00.872547 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:00.872549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:00.872780 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:00.872930 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:00.873585 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 14:51:00.873788 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":270,"threads_started":1}
I20260619 14:51:00.875177 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:00.857324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:38563 (local address 127.0.241.2:52253)
0619 14:51:00.858192 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:00.858229 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:00.860055 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:00.860827 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:00.860840 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:00.861323 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:00.861860 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:00.861902 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:00.862969 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:00.862982 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:00.864311 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:00.864328 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:00.865095 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:00.865104 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:00.865377 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:00.866104 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:00.866136 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:00.868582 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:00.871399 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:00.871408 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:00.871424 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:00.871860 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:00.872227 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:00.872232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:00.872236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:00.872413 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:00.872902 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:00.872909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:00.873224 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 14:51:00.873495 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 14:51:00.873969 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":310,"threads_started":1}
I20260619 14:51:00.876813 11612 heartbeater.cc:347] Connected to a master server at 127.0.241.62:38563
I20260619 14:51:00.877102 11612 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:00.877770 11612 heartbeater.cc:511] Master 127.0.241.62:38563 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
I20260619 14:51:00.879249 11289 ts_manager.cc:194] Registered new tserver with Master: 478742ce5ad94cc98f8a3ff1ea814802 (127.0.241.2:42885)
I20260619 14:51:00.880160 11289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:52253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:51:00.907389 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:38563
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:36507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:01.230252 11618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:01.230509 11618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:01.230600 11618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:01.237771 11618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:01.237882 11618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:01.237995 11618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:51:01.248220 11618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:36507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:38563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:51:01.249523 11618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:01.250782 11618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:01.264159 11623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:01.264279 11626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:01.264050 11624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:01.264940 11618 server_base.cc:1034] running on GCE node
I20260619 14:51:01.265573 11618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:01.266644 11618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:01.267870 11618 hybrid_clock.cc:648] HybridClock initialized: now 1781880661267825 us; error 46 us; skew 500 ppm
I20260619 14:51:01.268193 11618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:51:01 dist-test-slave-p3ml krb5kdc[11244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:01.271320 11618 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:51:01.272773 11618 webserver.cc:460] Webserver started at http://127.0.241.3:38729/ using document root <none> and password file <none>
I20260619 14:51:01.273456 11618 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:01.273600 11618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:01.273893 11618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:01.276803 11618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "a89a7df054724b7eac572a7e9ea487c8"
format_stamp: "Formatted at 2026-06-19 14:51:01 on dist-test-slave-p3ml"
server_key: "5f28226232556adffca918ff14f65cc2"
server_key_iv: "4f8adb5dd034ba29c49d7836d09a655d"
server_key_version: "encryptionkey@0"
I20260619 14:51:01.277535 11618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "a89a7df054724b7eac572a7e9ea487c8"
format_stamp: "Formatted at 2026-06-19 14:51:01 on dist-test-slave-p3ml"
server_key: "5f28226232556adffca918ff14f65cc2"
server_key_iv: "4f8adb5dd034ba29c49d7836d09a655d"
server_key_version: "encryptionkey@0"
I20260619 14:51:01.282500 11618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 14:51:01.286119 11633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:01.287407 11618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:51:01.287642 11618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "a89a7df054724b7eac572a7e9ea487c8"
format_stamp: "Formatted at 2026-06-19 14:51:01 on dist-test-slave-p3ml"
server_key: "5f28226232556adffca918ff14f65cc2"
server_key_iv: "4f8adb5dd034ba29c49d7836d09a655d"
server_key_version: "encryptionkey@0"
I20260619 14:51:01.287856 11618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:01.306061 11618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:01.310433 11618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:01.310743 11618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:01.311741 11618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:01.313043 11618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:01.313146 11618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:01.313299 11618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:01.313371 11618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:01.347689 11618 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:33427
I20260619 14:51:01.347779 11745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:33427 every 8 connection(s)
I20260619 14:51:01.349017 11618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:51:01.359481 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 11618
I20260619 14:51:01.359706 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:51:01.360239 964 external_mini_cluster.cc:1428] Setting key 75020848187f40f5d68332d53edc76e8
Jun 19 14:51:01 dist-test-slave-p3ml krb5kdc[11244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:51:01.364645 11478 heartbeater.cc:503] Master 127.0.241.62:38563 was elected leader, sending a full tablet report...
I20260619 14:51:01.369606 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.351486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:59507 (local address 127.0.241.62:38563)
0619 14:51:01.351702 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:01.351721 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:01.352908 (+ 1187us) server_negotiation.cc:407] Connection header received
0619 14:51:01.354637 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:01.354641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:01.354719 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:01.354808 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:01.356299 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.357019 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.358650 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.359036 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.359127 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.363131 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:01.363136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:01.363139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:01.363168 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:01.366561 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:01.367349 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:01.367354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:01.367360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:01.367532 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:01.367949 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:01.367954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:01.367957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:01.368221 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:01.368375 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:01.369035 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 14:51:01.369245 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 14:51:01.370477 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.352048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:38563 (local address 127.0.241.3:59507)
0619 14:51:01.352749 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:01.352789 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:01.354379 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:01.355006 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:01.355014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:01.355616 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:01.356148 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:01.356164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.357172 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.357176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:01.358506 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:01.358520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.359277 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.359286 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:01.359629 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.360415 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:01.360449 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:01.362894 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:01.366728 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:01.366736 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:01.366752 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:01.367218 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:01.367639 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:01.367644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:01.367648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:01.367830 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:01.368350 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:01.368358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:01.368684 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 14:51:01.368937 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 14:51:01.369420 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":270,"threads_started":1}
I20260619 14:51:01.371819 11746 heartbeater.cc:347] Connected to a master server at 127.0.241.62:38563
I20260619 14:51:01.372089 11746 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:01.372722 11746 heartbeater.cc:511] Master 127.0.241.62:38563 requested a full tablet report, sending...
I20260619 14:51:01.374084 11289 ts_manager.cc:194] Registered new tserver with Master: a89a7df054724b7eac572a7e9ea487c8 (127.0.241.3:33427)
I20260619 14:51:01.375144 11289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:59507
I20260619 14:51:01.375577 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:51:01.391414 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.379818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53236 (local address 127.0.241.62:38563)
0619 14:51:01.379959 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:01.379963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:01.380445 (+ 482us) server_negotiation.cc:407] Connection header received
0619 14:51:01.380548 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:01.380552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:01.380620 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:01.380700 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:01.381980 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.382776 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.384004 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.384412 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.384513 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.386770 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:01.386774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:01.386778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:01.386804 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:01.389141 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:01.389691 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:01.389695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:01.389697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:01.389807 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:01.390115 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:01.390118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:01.390120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:01.390380 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:01.390553 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:01.390882 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 14:51:01.391094 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:51:01.395128 11289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:51:01.401791 11289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:51:01.425063 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.413721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.0.241.3:33427)
0619 14:51:01.414619 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:01.414630 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:01.416654 (+ 2024us) server_negotiation.cc:407] Connection header received
0619 14:51:01.416781 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:01.416792 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:01.417095 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:01.417283 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:01.418658 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.420198 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.422916 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.423892 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.424039 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.424293 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:01.424540 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 14:51:01.424699 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":489,"threads_started":1}
I20260619 14:51:01.425164 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.413483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:33427 (local address 127.0.0.1:59946)
0619 14:51:01.416184 (+ 2701us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:01.416235 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:01.416443 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:01.417591 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:01.417599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:01.417651 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:01.418429 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:01.418450 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.420352 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.420358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:01.422735 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:01.422755 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.424092 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.424098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:01.424294 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.424350 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 14:51:01.424503 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 14:51:01.424697 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2436,"mutex_wait_us":36,"spinlock_wait_cycles":1408,"thread_start_us":5265,"threads_started":1}
I20260619 14:51:01.425958 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.413763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:42885 (local address 127.0.0.1:33210)
0619 14:51:01.416802 (+ 3039us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:01.416818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:01.416930 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:01.417585 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:01.417592 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:01.417618 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:01.418143 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:01.418153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.420068 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.420073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:01.422735 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:01.422755 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.425018 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.425026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:01.425240 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.425280 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 14:51:01.425375 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 14:51:01.425513 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2899,"mutex_wait_us":49,"thread_start_us":2026,"threads_started":1}
I20260619 14:51:01.426798 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.414305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.0.241.2:42885)
0619 14:51:01.415630 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:01.415638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:01.417043 (+ 1405us) server_negotiation.cc:407] Connection header received
0619 14:51:01.417153 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:01.417160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:01.417384 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:01.417584 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:01.418322 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.419914 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.422903 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.424517 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.424650 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.424849 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:01.426227 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0619 14:51:01.426402 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1138,"thread_start_us":942,"threads_started":1}
I20260619 14:51:01.428812 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.415291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33309 (local address 127.0.0.1:50440)
0619 14:51:01.419024 (+ 3733us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:01.419049 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:01.419207 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:01.420559 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:01.420568 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:01.420597 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:01.421266 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:01.421278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.422825 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.422833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:01.426133 (+ 3300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:01.426151 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.427890 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.427897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:01.428092 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.428139 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 14:51:01.428265 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 14:51:01.428392 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3604,"mutex_wait_us":41,"thread_start_us":1238,"threads_started":1}
I20260619 14:51:01.428803 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.415914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.0.241.1:33309)
0619 14:51:01.419899 (+ 3985us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:01.419908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:01.419937 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:51:01.420033 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:01.420042 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:01.420300 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:01.421440 (+ 1140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:01.421647 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.422673 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.426339 (+ 3666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.427680 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.427842 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.428090 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:01.428302 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 14:51:01.428463 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3804,"thread_start_us":3652,"threads_started":1}
I20260619 14:51:01.430444 11547 tablet_service.cc:1467] Processing CreateTablet for tablet 46e42f14e40943b6aedb4e0b2e8e68e2 (DEFAULT_TABLE table=test-table [id=87a67bd8195c47698240db6e8c5d02aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:01.431177 11681 tablet_service.cc:1467] Processing CreateTablet for tablet 46e42f14e40943b6aedb4e0b2e8e68e2 (DEFAULT_TABLE table=test-table [id=87a67bd8195c47698240db6e8c5d02aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:01.432073 11547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e42f14e40943b6aedb4e0b2e8e68e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:01.432675 11681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e42f14e40943b6aedb4e0b2e8e68e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:01.433311 11413 tablet_service.cc:1467] Processing CreateTablet for tablet 46e42f14e40943b6aedb4e0b2e8e68e2 (DEFAULT_TABLE table=test-table [id=87a67bd8195c47698240db6e8c5d02aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:01.434685 11413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e42f14e40943b6aedb4e0b2e8e68e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:01.444121 11764 tablet_bootstrap.cc:492] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802: Bootstrap starting.
I20260619 14:51:01.447539 11765 tablet_bootstrap.cc:492] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8: Bootstrap starting.
I20260619 14:51:01.447613 11764 tablet_bootstrap.cc:654] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:01.448961 11764 log.cc:824] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:01.449288 11766 tablet_bootstrap.cc:492] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177: Bootstrap starting.
I20260619 14:51:01.451186 11765 tablet_bootstrap.cc:654] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:01.452458 11765 log.cc:824] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:01.452888 11766 tablet_bootstrap.cc:654] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:01.453117 11764 tablet_bootstrap.cc:492] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802: No bootstrap required, opened a new log
I20260619 14:51:01.453434 11764 ts_tablet_manager.cc:1397] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 14:51:01.454160 11766 log.cc:824] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:01.455538 11765 tablet_bootstrap.cc:492] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8: No bootstrap required, opened a new log
I20260619 14:51:01.455803 11765 ts_tablet_manager.cc:1397] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 14:51:01.456984 11766 tablet_bootstrap.cc:492] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177: No bootstrap required, opened a new log
I20260619 14:51:01.457252 11766 ts_tablet_manager.cc:1397] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 14:51:01.464282 11764 raft_consensus.cc:348] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89a7df054724b7eac572a7e9ea487c8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33427 } } peers { permanent_uuid: "478742ce5ad94cc98f8a3ff1ea814802" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42885 } } peers { permanent_uuid: "e0926e6797794322ac541cb1e747c177" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33309 } }
I20260619 14:51:01.464731 11764 raft_consensus.cc:374] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:01.464870 11764 raft_consensus.cc:729] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478742ce5ad94cc98f8a3ff1ea814802, State: Initialized, Role: FOLLOWER
I20260619 14:51:01.465510 11764 consensus_queue.cc:260] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89a7df054724b7eac572a7e9ea487c8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33427 } } peers { permanent_uuid: "478742ce5ad94cc98f8a3ff1ea814802" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42885 } } peers { permanent_uuid: "e0926e6797794322ac541cb1e747c177" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33309 } }
I20260619 14:51:01.466473 11765 raft_consensus.cc:348] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89a7df054724b7eac572a7e9ea487c8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33427 } } peers { permanent_uuid: "478742ce5ad94cc98f8a3ff1ea814802" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42885 } } peers { permanent_uuid: "e0926e6797794322ac541cb1e747c177" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33309 } }
I20260619 14:51:01.466732 11612 heartbeater.cc:503] Master 127.0.241.62:38563 was elected leader, sending a full tablet report...
I20260619 14:51:01.466565 11766 raft_consensus.cc:348] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89a7df054724b7eac572a7e9ea487c8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33427 } } peers { permanent_uuid: "478742ce5ad94cc98f8a3ff1ea814802" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42885 } } peers { permanent_uuid: "e0926e6797794322ac541cb1e747c177" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33309 } }
I20260619 14:51:01.466982 11765 raft_consensus.cc:374] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:01.467095 11766 raft_consensus.cc:374] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:01.467145 11765 raft_consensus.cc:729] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a89a7df054724b7eac572a7e9ea487c8, State: Initialized, Role: FOLLOWER
I20260619 14:51:01.467243 11766 raft_consensus.cc:729] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0926e6797794322ac541cb1e747c177, State: Initialized, Role: FOLLOWER
I20260619 14:51:01.467465 11764 ts_tablet_manager.cc:1428] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 14:51:01.467936 11765 consensus_queue.cc:260] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89a7df054724b7eac572a7e9ea487c8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33427 } } peers { permanent_uuid: "478742ce5ad94cc98f8a3ff1ea814802" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42885 } } peers { permanent_uuid: "e0926e6797794322ac541cb1e747c177" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33309 } }
I20260619 14:51:01.468006 11766 consensus_queue.cc:260] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89a7df054724b7eac572a7e9ea487c8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33427 } } peers { permanent_uuid: "478742ce5ad94cc98f8a3ff1ea814802" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42885 } } peers { permanent_uuid: "e0926e6797794322ac541cb1e747c177" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33309 } }
I20260619 14:51:01.469069 11746 heartbeater.cc:503] Master 127.0.241.62:38563 was elected leader, sending a full tablet report...
I20260619 14:51:01.472119 11766 ts_tablet_manager.cc:1428] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260619 14:51:01.472664 11765 ts_tablet_manager.cc:1428] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260619 14:51:01.584466 11479 tablet.cc:2369] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:51:01.600612 11747 tablet.cc:2369] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:51:01.605844 11613 tablet.cc:2369] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:51:01.714828 11771 raft_consensus.cc:482] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:01.715188 11771 raft_consensus.cc:504] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89a7df054724b7eac572a7e9ea487c8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33427 } } peers { permanent_uuid: "478742ce5ad94cc98f8a3ff1ea814802" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42885 } } peers { permanent_uuid: "e0926e6797794322ac541cb1e747c177" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33309 } }
I20260619 14:51:01.716856 11771 leader_election.cc:288] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 478742ce5ad94cc98f8a3ff1ea814802 (127.0.241.2:42885), e0926e6797794322ac541cb1e747c177 (127.0.241.1:33309)
I20260619 14:51:01.724126 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.717426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:33017 (local address 127.0.241.2:42885)
0619 14:51:01.717634 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:01.717641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:01.718266 (+ 625us) server_negotiation.cc:407] Connection header received
0619 14:51:01.718460 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:01.718464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:01.718518 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:01.718624 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:01.719094 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.720058 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.722380 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.723118 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.723276 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.723369 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:01.723757 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 14:51:01.723905 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:51:01.724232 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.717334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33309 (local address 127.0.241.3:49571)
0619 14:51:01.717630 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:01.717671 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:01.717824 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:01.718219 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:01.718225 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:01.718252 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:01.718497 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:01.718506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.719877 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.719880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:01.722010 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:01.722024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.723548 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.723553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:01.723763 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.723793 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:51:01.723881 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:51:01.723997 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"spinlock_wait_cycles":17664}
I20260619 14:51:01.724287 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.717386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:42885 (local address 127.0.241.3:33017)
0619 14:51:01.718127 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:01.718166 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:01.718336 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:01.718656 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:01.718659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:01.718677 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:01.718943 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:01.718949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.720196 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.720200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:01.722225 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:01.722234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.723313 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.723319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:01.723525 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.723556 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:51:01.723704 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 14:51:01.723831 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":23,"spinlock_wait_cycles":22912,"thread_start_us":398,"threads_started":1}
I20260619 14:51:01.724793 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.717292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:49571 (local address 127.0.241.1:33309)
0619 14:51:01.717500 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:01.717506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:01.717785 (+ 279us) server_negotiation.cc:407] Connection header received
0619 14:51:01.717993 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:01.717999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:01.718083 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:01.718161 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:01.718636 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.719697 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.722177 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.723365 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.723531 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.723625 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:01.724465 (+ 840us) server_negotiation.cc:299] Negotiation successful
0619 14:51:01.724609 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:51:01.725401 11567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e42f14e40943b6aedb4e0b2e8e68e2" candidate_uuid: "a89a7df054724b7eac572a7e9ea487c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "478742ce5ad94cc98f8a3ff1ea814802" is_pre_election: true
I20260619 14:51:01.725597 11433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e42f14e40943b6aedb4e0b2e8e68e2" candidate_uuid: "a89a7df054724b7eac572a7e9ea487c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0926e6797794322ac541cb1e747c177" is_pre_election: true
I20260619 14:51:01.725919 11567 raft_consensus.cc:2454] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a89a7df054724b7eac572a7e9ea487c8 in term 0.
I20260619 14:51:01.726119 11433 raft_consensus.cc:2454] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a89a7df054724b7eac572a7e9ea487c8 in term 0.
I20260619 14:51:01.726577 11634 leader_election.cc:302] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 478742ce5ad94cc98f8a3ff1ea814802, a89a7df054724b7eac572a7e9ea487c8; no voters:
I20260619 14:51:01.726999 11771 raft_consensus.cc:2781] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:51:01.727154 11771 raft_consensus.cc:482] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:51:01.727259 11771 raft_consensus.cc:3037] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:01.729405 11771 raft_consensus.cc:504] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89a7df054724b7eac572a7e9ea487c8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33427 } } peers { permanent_uuid: "478742ce5ad94cc98f8a3ff1ea814802" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42885 } } peers { permanent_uuid: "e0926e6797794322ac541cb1e747c177" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33309 } }
I20260619 14:51:01.730003 11771 leader_election.cc:288] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [CANDIDATE]: Term 1 election: Requested vote from peers 478742ce5ad94cc98f8a3ff1ea814802 (127.0.241.2:42885), e0926e6797794322ac541cb1e747c177 (127.0.241.1:33309)
I20260619 14:51:01.730438 11567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e42f14e40943b6aedb4e0b2e8e68e2" candidate_uuid: "a89a7df054724b7eac572a7e9ea487c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "478742ce5ad94cc98f8a3ff1ea814802"
I20260619 14:51:01.730518 11433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e42f14e40943b6aedb4e0b2e8e68e2" candidate_uuid: "a89a7df054724b7eac572a7e9ea487c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0926e6797794322ac541cb1e747c177"
I20260619 14:51:01.730636 11567 raft_consensus.cc:3037] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:01.730718 11433 raft_consensus.cc:3037] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:01.732801 11567 raft_consensus.cc:2454] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a89a7df054724b7eac572a7e9ea487c8 in term 1.
I20260619 14:51:01.733193 11634 leader_election.cc:302] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 478742ce5ad94cc98f8a3ff1ea814802, a89a7df054724b7eac572a7e9ea487c8; no voters:
I20260619 14:51:01.733456 11771 raft_consensus.cc:2781] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:01.733749 11433 raft_consensus.cc:2454] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a89a7df054724b7eac572a7e9ea487c8 in term 1.
I20260619 14:51:01.734149 11771 raft_consensus.cc:686] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 1 LEADER]: Becoming Leader. State: Replica: a89a7df054724b7eac572a7e9ea487c8, State: Running, Role: LEADER
I20260619 14:51:01.734917 11771 consensus_queue.cc:237] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89a7df054724b7eac572a7e9ea487c8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33427 } } peers { permanent_uuid: "478742ce5ad94cc98f8a3ff1ea814802" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42885 } } peers { permanent_uuid: "e0926e6797794322ac541cb1e747c177" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33309 } }
I20260619 14:51:01.739225 11288 catalog_manager.cc:5495] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a89a7df054724b7eac572a7e9ea487c8 (127.0.241.3). New cstate: current_term: 1 leader_uuid: "a89a7df054724b7eac572a7e9ea487c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a89a7df054724b7eac572a7e9ea487c8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "478742ce5ad94cc98f8a3ff1ea814802" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0926e6797794322ac541cb1e747c177" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33309 } health_report { overall_health: UNKNOWN } } }
I20260619 14:51:01.789186 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.784024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.0.241.3:33427)
0619 14:51:01.784192 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:01.784197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:01.784213 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:51:01.784316 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:01.784319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:01.784390 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:01.784513 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:01.784836 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.785725 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.787100 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.787530 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.787628 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.788088 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:01.788594 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:01.788714 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:01.788792 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 14:51:01.788854 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:51:01.796810 11433 raft_consensus.cc:1261] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177 [term 1 FOLLOWER]: Refusing update from remote peer a89a7df054724b7eac572a7e9ea487c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:01.796825 11567 raft_consensus.cc:1261] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802 [term 1 FOLLOWER]: Refusing update from remote peer a89a7df054724b7eac572a7e9ea487c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:01.797751 11774 consensus_queue.cc:1035] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0926e6797794322ac541cb1e747c177" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:01.798179 11771 consensus_queue.cc:1035] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "478742ce5ad94cc98f8a3ff1ea814802" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:01.810144 11777 mvcc.cc:205] Tried to move back new op lower bound from 7298583190710099968 to 7298583190470008832. Current Snapshot: MvccSnapshot[applied={T|T < 7298583190710099968}]
I20260619 14:51:01.823705 11289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:01.824012 11289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:01.827549 11289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a08f5b33ff9e495e91a8b357b01f1d0d: Sending DeleteTablet for 3 replicas of tablet 46e42f14e40943b6aedb4e0b2e8e68e2
I20260619 14:51:01.828722 11681 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e42f14e40943b6aedb4e0b2e8e68e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:01 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:59946
I20260619 14:51:01.828716 11547 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e42f14e40943b6aedb4e0b2e8e68e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:01 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:33210
I20260619 14:51:01.828749 11413 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e42f14e40943b6aedb4e0b2e8e68e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:01 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:50440
I20260619 14:51:01.830937 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 11350
I20260619 14:51:01.841009 11788 tablet_replica.cc:330] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8: stopping tablet replica
I20260619 14:51:01.841405 11788 raft_consensus.cc:2229] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:51:01.841990 11788 raft_consensus.cc:2258] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:01.845482 11788 ts_tablet_manager.cc:1905] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:01.848624 11789 tablet_replica.cc:330] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802: stopping tablet replica
I20260619 14:51:01.849184 11789 raft_consensus.cc:2229] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:01.849812 11789 raft_consensus.cc:2258] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:01.851565 11787 tablet_replica.cc:330] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177: stopping tablet replica
I20260619 14:51:01.853727 11788 ts_tablet_manager.cc:1918] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:51:01.854063 11788 log.cc:1196] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/46e42f14e40943b6aedb4e0b2e8e68e2
I20260619 14:51:01.855649 11789 ts_tablet_manager.cc:1905] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:01.859705 11788 ts_tablet_manager.cc:1939] T 46e42f14e40943b6aedb4e0b2e8e68e2 P a89a7df054724b7eac572a7e9ea487c8: Deleting consensus metadata
I20260619 14:51:01.861267 11787 raft_consensus.cc:2229] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177 [term 1 FOLLOWER]: Raft consensus shutting down.
Jun 19 14:51:01 dist-test-slave-p3ml krb5kdc[11244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:51:01.861640 11274 catalog_manager.cc:4841] TS a89a7df054724b7eac572a7e9ea487c8 (127.0.241.3:33427): tablet 46e42f14e40943b6aedb4e0b2e8e68e2 (table test-table [id=87a67bd8195c47698240db6e8c5d02aa]) successfully deleted
I20260619 14:51:01.861864 11787 raft_consensus.cc:2258] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:01.867121 11787 ts_tablet_manager.cc:1905] T 46e42f14e40943b6aedb4e0b2e8e68e2 P e0926e6797794322ac541cb1e747c177: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:01.870272 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.832449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.0.241.1:33309)
0619 14:51:01.832612 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:01.832618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:01.850771 (+ 18153us) server_negotiation.cc:407] Connection header received
0619 14:51:01.850969 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:01.850975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:01.851092 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:01.851238 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:01.853165 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.854314 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.856230 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.856874 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.856983 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:01.862669 (+ 5686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:01.862683 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:01.862701 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:01.862750 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:01.866758 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:01.867702 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:01.867711 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:01.867715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:01.867884 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:01.868358 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:01.868363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:01.868368 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:01.868831 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:01.869109 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:01.869671 (+ 562us) server_negotiation.cc:299] Negotiation successful
0619 14:51:01.869899 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:51:01.873194 11473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:51:01.870211 11789 ts_tablet_manager.cc:1918] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:51:01.875226 11789 log.cc:1196] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/46e42f14e40943b6aedb4e0b2e8e68e2
I20260619 14:51:01.876124 11789 ts_tablet_manager.cc:1939] T 46e42f14e40943b6aedb4e0b2e8e68e2 P 478742ce5ad94cc98f8a3ff1ea814802: Deleting consensus metadata
I20260619 14:51:01.878636 11273 catalog_manager.cc:4841] TS 478742ce5ad94cc98f8a3ff1ea814802 (127.0.241.2:42885): tablet 46e42f14e40943b6aedb4e0b2e8e68e2 (table test-table [id=87a67bd8195c47698240db6e8c5d02aa]) successfully deleted
I20260619 14:51:01.972101 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 11350
I20260619 14:51:01.989769 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 11484
W20260619 14:51:01.989847 11276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.241.1:33309 (error 108)
I20260619 14:51:01.991994 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.991142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33309 (local address 127.0.0.1:50458)
0619 14:51:01.991617 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:01.991758 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":200,"threads_started":1}
W20260619 14:51:01.992257 11276 catalog_manager.cc:4584] TS e0926e6797794322ac541cb1e747c177 (127.0.241.1:33309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46e42f14e40943b6aedb4e0b2e8e68e2: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33309: connect: Connection refused (error 111)
Jun 19 14:51:01 dist-test-slave-p3ml krb5kdc[11244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:51:02.006978 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:01.990872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.0.241.2:42885)
0619 14:51:01.991043 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:01.991049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:01.991070 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:51:01.991140 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:01.991145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:01.991250 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:01.991357 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:01.993111 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.994247 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.995483 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:01.995874 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:01.996016 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:02.000300 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:02.000318 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:02.000339 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:02.000387 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:02.004057 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:02.004725 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:02.004735 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:02.004740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:02.004897 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:02.005336 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:02.005341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:02.005347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:02.005817 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:02.006111 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:02.006387 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 14:51:02.006603 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:51:02.007697 11607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:51:02.046093 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:02.045525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33309 (local address 127.0.0.1:50466)
0619 14:51:02.045791 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:02.045898 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 14:51:02.047339 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:02.046908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33309 (local address 127.0.0.1:50476)
0619 14:51:02.047093 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:02.047184 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 14:51:02.096988 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 11484
I20260619 14:51:02.108166 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:02.107592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33309 (local address 127.0.0.1:50486)
0619 14:51:02.107843 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:02.107973 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 14:51:02.109375 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:02.108933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33309 (local address 127.0.0.1:50498)
0619 14:51:02.109105 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:02.109203 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 14:51:02.114710 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 11618
Jun 19 14:51:02 dist-test-slave-p3ml krb5kdc[11244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:51:02.130892 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:02.115499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.0.241.3:33427)
0619 14:51:02.115675 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:02.115682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:02.115701 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:51:02.115918 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:02.115923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:02.115999 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:02.116110 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:02.117389 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:02.118222 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:02.119580 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:02.120021 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:02.120168 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:02.124801 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:02.124812 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:02.124831 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:02.124876 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:02.128059 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:02.128860 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:02.128866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:02.128868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:02.129015 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:02.129418 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:02.129422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:02.129424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:02.129756 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:02.129979 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:02.130375 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 14:51:02.130519 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:51:02.131831 11741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:51:02.189709 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:02.189243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33309 (local address 127.0.0.1:50504)
0619 14:51:02.189461 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:02.189552 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 14:51:02.190732 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:02.190383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33309 (local address 127.0.0.1:50510)
0619 14:51:02.190548 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:02.190609 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 14:51:02.253705 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 11618
I20260619 14:51:02.273521 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 11257
I20260619 14:51:02.274751 11319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:51:02.379832 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 11257
I20260619 14:51:02.379871 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:02.379203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:33309 (local address 127.0.0.1:50520)
0619 14:51:02.379601 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:02.379712 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:33309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
2026-06-19T14:51:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4406 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:02 dist-test-slave-p3ml krb5kdc[11799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:02 dist-test-slave-p3ml krb5kdc[11799](info): set up 2 sockets
Jun 19 14:51:02 dist-test-slave-p3ml krb5kdc[11799](info): commencing operation
krb5kdc: starting...
W20260619 14:51:03.508333 964 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:51:03 dist-test-slave-p3ml krb5kdc[11799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.241.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.241.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/rangeradmin_127.0.241.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/rangeradmin_127.0.241.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.241.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.241.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/rangerlookup_127.0.241.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/rangerlookup_127.0.241.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.241.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.241.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/HTTP_127.0.241.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/HTTP_127.0.241.20@KRBTEST.COM.keytab.
I20260619 14:51:03.824368 964 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskydHOLK/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/postgres -l logfile start
2026-06-19 14:51:06.239 UTC [11829] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:51:06.239 UTC [11829] LOG: listening on IPv4 address "127.0.241.20", port 34999
2026-06-19 14:51:06.242 UTC [11829] LOG: listening on Unix socket "/tmp/.s.PGSQL.34999"
2026-06-19 14:51:06.263 UTC [11832] LOG: database system was shut down at 2026-06-19 14:51:04 UTC
2026-06-19 14:51:06.267 UTC [11829] LOG: database system is ready to accept connections
I20260619 14:51:07.236037 964 mini_postgres.cc:96] Postgres bound to 34999
2026-06-19 14:51:07.253 UTC [11840] FATAL: database "slave" does not exist
127.0.241.20:34999 - accepting connections
I20260619 14:51:07.254093 964 mini_ranger.cc:161] Starting Ranger...
I20260619 14:51:07.279990 964 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:51:07.747323 964 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:51:07.747500 964 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-admin
2026-06-19 14:51:08,038 [I] DB FLAVOR :POSTGRES
2026-06-19 14:51:08,038 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:51:08,039 [I] Checking connection..
2026-06-19 14:51:08,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:51:08,305 [I] Checking connection passed.
2026-06-19 14:51:08,306 [I] --------- Verifying version history table ---------
2026-06-19 14:51:08,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:51:08,562 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:51:08,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:51:08,803 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:51:08,803 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:51:08,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:51:09.043 UTC [11904] WARNING: there is no transaction in progress
2026-06-19 14:51:09,050 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:51:09,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:51:09,305 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:51:09,305 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:51:09,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:51:09,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:51:09,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p3ml.c.gcp-upstream.internal','N') ;"
2026-06-19 14:51:10,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:51:10,327 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:51:10,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:51:10,573 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:51:10,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:51:10,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:51:11,091 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:51:11,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:51:12.011 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.027 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.043 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.058 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.074 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.092 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.155 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.161 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.175 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.185 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.194 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.208 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.217 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.223 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.230 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12.236 UTC [12012] WARNING: there is no transaction in progress
2026-06-19 14:51:12,832 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:51:12,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:51:13,052 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:51:13,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:51:13,309 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:51:13,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:51:13,571 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:51:13,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:51:13,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:51:14,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p3ml.c.gcp-upstream.internal';"
2026-06-19 14:51:14,291 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:51:14,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:51:14,549 [I] DB_PATCHES have already been applied
I20260619 14:51:14.555895 964 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-admin:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskydHOLK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:51:14.555994 964 mini_ranger.cc:193] Using host: 127.0.241.20
I20260619 14:51:14.570730 964 mini_ranger.cc:227] Ranger admin URL: http://127.0.241.20:46235
Jun 19, 2026 2:51:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:51:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:51:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:51:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:51:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:51:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:51:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:51:15.726884 964 mini_ranger.cc:160] Time spent starting Ranger: real 8.473s user 0.000s sys 0.039s
Jun 19, 2026 2:51:34 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:51:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 2:51:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:51:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:51:34 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:51:35 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:51:36.185966 964 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.241.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.241.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/rangerkms_127.0.241.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/rangerkms_127.0.241.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.241.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.241.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.241.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/HTTP_127.0.241.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.241.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/HTTP_127.0.241.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:51:36 dist-test-slave-p3ml krb5kdc[11799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880696, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:51:36.347779 964 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:51:36.368899 964 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:51:37.172099 964 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:51:37.172271 964 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-kms
I20260619 14:51:37.172400 964 mini_ranger_kms.cc:227] Using postgres at 127.0.241.20:34999
2026-06-19 14:51:37,617 [I] DB FLAVOR :POSTGRES
2026-06-19 14:51:37,617 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:51:37,617 [I] Checking connection
2026-06-19 14:51:37,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:51:37,853 [I] connection success
2026-06-19 14:51:37,854 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:51:37,854 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:51:37,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:51:38,121 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:51:38,121 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:51:38,121 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:51:38,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:34999/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:51:38,383 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:51:38.835490 964 mini_ranger_kms.cc:310] Created kms service
I20260619 14:51:39.003412 964 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:51:39.044867 964 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:51:39 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880696, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:51:39.453804 964 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:51:39.454155 964 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-kms:/tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:51:39.454223 964 mini_ranger_kms.cc:240] Using host: 127.0.241.20
I20260619 14:51:39.471700 964 mini_ranger_kms.cc:276] Ranger KMS PID: 12192
I20260619 14:51:39.471911 964 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.241.20:60077
Jun 19, 2026 2:51:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:51:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:51:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:51:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:51:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:51:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 14:51:41.744652 964 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.397s user 0.122s sys 0.337s
I20260619 14:51:41.744956 964 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:51:41.745178 964 mini_ranger_kms.cc:401] 127.0.241.20:60077/kms/v1/keys
Jun 19 14:51:41 dist-test-slave-p3ml krb5kdc[11799](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880701, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.241.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 14:51:42 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880701, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.241.20@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
Jun 19 14:51:42 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880701, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.241.20@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
Jun 19 14:51:44 dist-test-slave-p3ml krb5kdc[11799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:51:44.821578 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33007
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:33007
--ranger_config_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.241.20:60077/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:45.159065 12243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:45.159406 12243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:45.159572 12243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:45.167222 12243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:45.167364 12243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:45.167536 12243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:45.167604 12243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:45.167688 12243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:45.178129 12243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.241.20:60077/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:33007
--ranger_config_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:51:45.179411 12243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:45.180737 12243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:45.193758 12251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:45.193799 12248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:45.193821 12249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:45.195574 12243 server_base.cc:1034] running on GCE node
I20260619 14:51:45.196480 12243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:45.198239 12243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:45.199505 12243 hybrid_clock.cc:648] HybridClock initialized: now 1781880705199444 us; error 51 us; skew 500 ppm
I20260619 14:51:45.199891 12243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:51:45 dist-test-slave-p3ml krb5kdc[11799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:45.203693 12243 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:51:45.205206 12243 webserver.cc:460] Webserver started at http://127.0.241.62:41681/ using document root <none> and password file <none>
I20260619 14:51:45.205864 12243 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:45.205996 12243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:45.206295 12243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:51:45 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:51:45.264372 12243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "878bea2c186345fd9dd144a7392a1928"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-p3ml"
server_key: "9859a84ec779c163be8c734ddab3a0c7"
server_key_iv: "dd044d582162a3f195010d983ad4f7ec"
server_key_version: "kuduclusterkey@0"
I20260619 14:51:45.265372 12243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "878bea2c186345fd9dd144a7392a1928"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-p3ml"
server_key: "9859a84ec779c163be8c734ddab3a0c7"
server_key_iv: "dd044d582162a3f195010d983ad4f7ec"
server_key_version: "kuduclusterkey@0"
I20260619 14:51:45.272053 12243 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 14:51:45.288640 12258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:45.290683 12243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:51:45.290907 12243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "878bea2c186345fd9dd144a7392a1928"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-p3ml"
server_key: "9859a84ec779c163be8c734ddab3a0c7"
server_key_iv: "dd044d582162a3f195010d983ad4f7ec"
server_key_version: "kuduclusterkey@0"
I20260619 14:51:45.291193 12243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:45.337747 12243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:45.342101 12243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:45.342424 12243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:45.363106 12243 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:33007
I20260619 14:51:45.363125 12309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:33007 every 8 connection(s)
I20260619 14:51:45.364588 12243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:45.368467 12310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:45.371316 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 12243
I20260619 14:51:45.371555 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:51:45.372371 964 external_mini_cluster.cc:1428] Setting key b2738264ed53eb4994a65967f0998aed
I20260619 14:51:45.383883 12310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928: Bootstrap starting.
I20260619 14:51:45.389017 12310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:45.390553 12310 log.cc:824] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:45.394462 12310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928: No bootstrap required, opened a new log
I20260619 14:51:45.404307 12310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878bea2c186345fd9dd144a7392a1928" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33007 } }
Jun 19 14:51:45 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:51:45.404721 12310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:45.404863 12310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878bea2c186345fd9dd144a7392a1928, State: Initialized, Role: FOLLOWER
I20260619 14:51:45.405589 12310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878bea2c186345fd9dd144a7392a1928" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33007 } }
I20260619 14:51:45.405864 12310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:45.406023 12310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:45.406260 12310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:45.410506 12310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878bea2c186345fd9dd144a7392a1928" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33007 } }
I20260619 14:51:45.411198 12310 leader_election.cc:302] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 878bea2c186345fd9dd144a7392a1928; no voters:
I20260619 14:51:45.412467 12310 leader_election.cc:288] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:45.412573 12315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:45.416193 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:45.374508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.0.241.62:33007)
0619 14:51:45.389740 (+ 15232us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:45.389760 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:45.389810 (+ 50us) server_negotiation.cc:407] Connection header received
0619 14:51:45.392687 (+ 2877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:45.392714 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:45.393187 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:45.393618 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:45.395290 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.397607 (+ 2317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.399639 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.400673 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.400865 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:45.406067 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:45.406076 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:45.406094 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:45.406142 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:45.411031 (+ 4889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:45.411868 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:45.411877 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:45.411886 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:45.412336 (+ 450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:45.412812 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:45.412817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:45.412821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:45.413496 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:45.413762 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:45.414456 (+ 694us) server_negotiation.cc:299] Negotiation successful
0619 14:51:45.414851 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14914,"thread_start_us":14687,"threads_started":1}
I20260619 14:51:45.416780 12315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [term 1 LEADER]: Becoming Leader. State: Replica: 878bea2c186345fd9dd144a7392a1928, State: Running, Role: LEADER
I20260619 14:51:45.417613 12315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878bea2c186345fd9dd144a7392a1928" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33007 } }
I20260619 14:51:45.420820 12310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:45.422235 12317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "878bea2c186345fd9dd144a7392a1928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878bea2c186345fd9dd144a7392a1928" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33007 } } }
I20260619 14:51:45.422571 12317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:45.423004 12316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 878bea2c186345fd9dd144a7392a1928. Latest consensus state: current_term: 1 leader_uuid: "878bea2c186345fd9dd144a7392a1928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878bea2c186345fd9dd144a7392a1928" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33007 } } }
I20260619 14:51:45.423416 12316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:45.426968 12320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:45.432534 12320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:45.454124 12320 catalog_manager.cc:1347] Generated new cluster ID: 048eb0a3800e4f148859bb8c877799af
I20260619 14:51:45.454249 12320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:45.467747 12320 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:45.470063 12320 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:45.476819 12320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928: Generated new TSK 0
I20260619 14:51:45.477928 12320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:45.484794 12310 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:51:46.360158 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.241.20:60077/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33007
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:51:46 dist-test-slave-p3ml krb5kdc[11799](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 14:51:46.789901 12355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:46.790336 12355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:46.790498 12355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:46.801944 12355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:46.802136 12355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:46.802350 12355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:51:46.817247 12355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.241.20:60077/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:51:46.819077 12355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:46.820716 12355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:46.839176 12365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:46.840814 12366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:46.844008 12368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:46.844512 12355 server_base.cc:1034] running on GCE node
I20260619 14:51:46.845189 12355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:46.846200 12355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:46.847493 12355 hybrid_clock.cc:648] HybridClock initialized: now 1781880706847389 us; error 58 us; skew 500 ppm
I20260619 14:51:46.847868 12355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:51:46 dist-test-slave-p3ml krb5kdc[11799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:46.852070 12355 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:51:46.854537 12355 webserver.cc:460] Webserver started at http://127.0.241.1:36049/ using document root <none> and password file <none>
I20260619 14:51:46.855621 12355 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:46.855792 12355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:46.856609 12355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:51:46 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:51:46.879022 12355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "86ff531f5b2f4294ba11570565285a2a"
format_stamp: "Formatted at 2026-06-19 14:51:46 on dist-test-slave-p3ml"
server_key: "2f70a2bdb61492e1367f7c04734b8a59"
server_key_iv: "3b2de31ed900a2b478e2d5f7771048ea"
server_key_version: "kuduclusterkey@0"
I20260619 14:51:46.880234 12355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "86ff531f5b2f4294ba11570565285a2a"
format_stamp: "Formatted at 2026-06-19 14:51:46 on dist-test-slave-p3ml"
server_key: "2f70a2bdb61492e1367f7c04734b8a59"
server_key_iv: "3b2de31ed900a2b478e2d5f7771048ea"
server_key_version: "kuduclusterkey@0"
I20260619 14:51:46.903213 12355 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.013s sys 0.004s
I20260619 14:51:46.934839 12375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:46.937211 12355 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.013s
I20260619 14:51:46.937487 12355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "86ff531f5b2f4294ba11570565285a2a"
format_stamp: "Formatted at 2026-06-19 14:51:46 on dist-test-slave-p3ml"
server_key: "2f70a2bdb61492e1367f7c04734b8a59"
server_key_iv: "3b2de31ed900a2b478e2d5f7771048ea"
server_key_version: "kuduclusterkey@0"
I20260619 14:51:46.938014 12355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:46.962505 12355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:46.968057 12355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:46.968420 12355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:46.970798 12355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:46.974007 12355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:46.974100 12355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:46.974220 12355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:46.974277 12355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:47.014590 12355 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:41981
I20260619 14:51:47.014608 12487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:41981 every 8 connection(s)
I20260619 14:51:47.015964 12355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:51:47.019750 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 12355
I20260619 14:51:47.019945 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:51:47.020747 964 external_mini_cluster.cc:1428] Setting key 055a88979c3eb8cb1c55562e5961a073
Jun 19 14:51:47 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:51:47.066706 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.019265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:47807 (local address 127.0.241.62:33007)
0619 14:51:47.047614 (+ 28349us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:47.047620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:47.047639 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:51:47.047699 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:47.047704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:47.047803 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:47.050046 (+ 2243us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:47.050207 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.051284 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.053542 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.054142 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.054254 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.058667 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:47.058674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:47.058678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:47.058737 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:47.062085 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:47.063365 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:47.063370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:47.063373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:47.063542 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:47.064089 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:47.064093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:47.064096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:47.064415 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:47.064538 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:47.066141 (+ 1603us) server_negotiation.cc:299] Negotiation successful
0619 14:51:47.066365 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28195,"thread_start_us":28605,"threads_started":1}
I20260619 14:51:47.068645 12491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.019687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33007 (local address 127.0.241.1:47807)
0619 14:51:47.044525 (+ 24838us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:47.044578 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:47.046859 (+ 2281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:47.048314 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:47.048327 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:47.048935 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:47.049859 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:47.049884 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.051492 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.051497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:47.053361 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:47.053376 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.054537 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.054543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:47.054860 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.055698 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:47.055741 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:47.058398 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:47.062238 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:47.062246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:47.062264 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:47.063218 (+ 954us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:47.063672 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:47.063678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:47.063682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:47.063952 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:47.064617 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:47.064625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:47.065037 (+ 412us) client_negotiation.cc:769] Sending connection context
0619 14:51:47.065357 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 14:51:47.067360 (+ 2003us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24466,"thread_start_us":24407,"threads_started":1}
I20260619 14:51:47.071009 12488 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33007
I20260619 14:51:47.071328 12488 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:47.072247 12488 heartbeater.cc:511] Master 127.0.241.62:33007 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
I20260619 14:51:47.074678 12275 ts_manager.cc:194] Registered new tserver with Master: 86ff531f5b2f4294ba11570565285a2a (127.0.241.1:41981)
I20260619 14:51:47.076782 12275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:47807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Jun 19 14:51:47 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:51:47.119797 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.241.20:60077/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33007
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:51:47 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
W20260619 14:51:47.587383 12494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:47.587877 12494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:47.588078 12494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:47.598690 12494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:47.598912 12494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:47.599164 12494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:51:47.614531 12494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.241.20:60077/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:51:47.616298 12494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:47.618052 12494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:47.649158 12501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:47.649829 12502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:47.669134 12504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:47.670200 12494 server_base.cc:1034] running on GCE node
I20260619 14:51:47.671557 12494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:47.674010 12494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:47.675308 12494 hybrid_clock.cc:648] HybridClock initialized: now 1781880707675233 us; error 72 us; skew 500 ppm
I20260619 14:51:47.675721 12494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:51:47 dist-test-slave-p3ml krb5kdc[11799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:47.680087 12494 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:51:47.692945 12494 webserver.cc:460] Webserver started at http://127.0.241.2:45017/ using document root <none> and password file <none>
I20260619 14:51:47.694388 12494 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:47.694558 12494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:47.694998 12494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:51:47 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:51:47.716560 12494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "84aa514155af43e8adf5a45374abf171"
format_stamp: "Formatted at 2026-06-19 14:51:47 on dist-test-slave-p3ml"
server_key: "d242c6555ad2d9475c85aa1e3ccd6521"
server_key_iv: "d2ef19b19bc7db0f96fb79926d8d5295"
server_key_version: "kuduclusterkey@0"
I20260619 14:51:47.717592 12494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "84aa514155af43e8adf5a45374abf171"
format_stamp: "Formatted at 2026-06-19 14:51:47 on dist-test-slave-p3ml"
server_key: "d242c6555ad2d9475c85aa1e3ccd6521"
server_key_iv: "d2ef19b19bc7db0f96fb79926d8d5295"
server_key_version: "kuduclusterkey@0"
I20260619 14:51:47.725755 12494 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260619 14:51:47.742473 12518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:47.744566 12494 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:51:47.744766 12494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "84aa514155af43e8adf5a45374abf171"
format_stamp: "Formatted at 2026-06-19 14:51:47 on dist-test-slave-p3ml"
server_key: "d242c6555ad2d9475c85aa1e3ccd6521"
server_key_iv: "d2ef19b19bc7db0f96fb79926d8d5295"
server_key_version: "kuduclusterkey@0"
I20260619 14:51:47.745059 12494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:47.763393 12494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:47.768175 12494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:47.768515 12494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:47.770097 12494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:47.771674 12494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:47.771759 12494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:47.771945 12494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:47.772004 12494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:47.816289 12494 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:34095
I20260619 14:51:47.816303 12630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:34095 every 8 connection(s)
I20260619 14:51:47.817595 12494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:51:47.818940 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 12494
I20260619 14:51:47.819190 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:51:47.819856 964 external_mini_cluster.cc:1428] Setting key f868ec7f70f8f36d76af803416e74f0b
Jun 19 14:51:47 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
I20260619 14:51:47.845903 12633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.821903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:50879 (local address 127.0.241.62:33007)
0619 14:51:47.822436 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:47.822443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:47.825382 (+ 2939us) server_negotiation.cc:407] Connection header received
0619 14:51:47.825970 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:47.825976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:47.826053 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:47.826150 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:47.828227 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.829394 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.832316 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.832903 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.833000 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.837842 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:47.837848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:47.837854 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:47.837893 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:47.841483 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:47.842511 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:47.842518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:47.842522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:47.842685 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:47.843327 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:47.843333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:47.843337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:47.843809 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:47.844001 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:47.845131 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0619 14:51:47.845368 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":230,"threads_started":1}
I20260619 14:51:47.846577 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.822197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33007 (local address 127.0.241.2:50879)
0619 14:51:47.823075 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:47.823127 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:47.825557 (+ 2430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:47.826599 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:47.826611 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:47.827204 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:47.828019 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:47.828043 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.829580 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.829585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:47.832117 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:47.832131 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.833344 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.833350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:47.833631 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.834487 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:47.834515 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:47.837570 (+ 3055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:47.841648 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:47.841659 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:47.841679 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:47.842350 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:47.842821 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:47.842829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:47.842834 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:47.843133 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:47.843984 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:47.843996 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:47.844511 (+ 515us) client_negotiation.cc:769] Sending connection context
0619 14:51:47.844965 (+ 454us) client_negotiation.cc:240] Negotiation successful
0619 14:51:47.845384 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":376,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
I20260619 14:51:47.848258 12631 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33007
I20260619 14:51:47.848584 12631 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:47.849527 12631 heartbeater.cc:511] Master 127.0.241.62:33007 requested a full tablet report, sending...
I20260619 14:51:47.851486 12275 ts_manager.cc:194] Registered new tserver with Master: 84aa514155af43e8adf5a45374abf171 (127.0.241.2:34095)
I20260619 14:51:47.852645 12275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:50879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:51:47.876552 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.241.20:60077/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33007
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:51:48.080466 12488 heartbeater.cc:503] Master 127.0.241.62:33007 was elected leader, sending a full tablet report...
W20260619 14:51:48.203855 12637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:48.204154 12637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:48.204308 12637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:48.211686 12637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:48.211824 12637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:48.212001 12637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:51:48.221964 12637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.241.20:60077/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:33007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:51:48.223245 12637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:48.224602 12637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:48.238307 12642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:48.238343 12643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:48.238363 12645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:48.239722 12637 server_base.cc:1034] running on GCE node
I20260619 14:51:48.240429 12637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:48.241433 12637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:48.242689 12637 hybrid_clock.cc:648] HybridClock initialized: now 1781880708242607 us; error 52 us; skew 500 ppm
I20260619 14:51:48.243022 12637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:51:48 dist-test-slave-p3ml krb5kdc[11799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:48.246572 12637 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:51:48.248026 12637 webserver.cc:460] Webserver started at http://127.0.241.3:33365/ using document root <none> and password file <none>
I20260619 14:51:48.248669 12637 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:48.249221 12637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:48.249541 12637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:51:48 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:51:48.264935 12637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "d99dc864df914e9785066ebf63831fa7"
format_stamp: "Formatted at 2026-06-19 14:51:48 on dist-test-slave-p3ml"
server_key: "901e708ba533fd35e5cc0fbdf565316b"
server_key_iv: "b977ef1856f061af544e4cca505a98a9"
server_key_version: "kuduclusterkey@0"
I20260619 14:51:48.265758 12637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "d99dc864df914e9785066ebf63831fa7"
format_stamp: "Formatted at 2026-06-19 14:51:48 on dist-test-slave-p3ml"
server_key: "901e708ba533fd35e5cc0fbdf565316b"
server_key_iv: "b977ef1856f061af544e4cca505a98a9"
server_key_version: "kuduclusterkey@0"
I20260619 14:51:48.271056 12637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:51:48.285481 12652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:48.286960 12637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:51:48.287120 12637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "d99dc864df914e9785066ebf63831fa7"
format_stamp: "Formatted at 2026-06-19 14:51:48 on dist-test-slave-p3ml"
server_key: "901e708ba533fd35e5cc0fbdf565316b"
server_key_iv: "b977ef1856f061af544e4cca505a98a9"
server_key_version: "kuduclusterkey@0"
I20260619 14:51:48.287369 12637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:48.313082 12637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:48.317189 12637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:48.317487 12637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:48.318555 12637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:48.319970 12637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:48.320041 12637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:48.320194 12637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:48.320251 12637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:48.364225 12637 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:45687
I20260619 14:51:48.364465 12764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:45687 every 8 connection(s)
I20260619 14:51:48.365746 12637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:51:48.370512 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 12637
I20260619 14:51:48.370702 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:51:48.371304 964 external_mini_cluster.cc:1428] Setting key ba345aa18f19d71fcfe62597df4f1b41
Jun 19 14:51:48 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:51:48.396478 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.371793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:46971 (local address 127.0.241.62:33007)
0619 14:51:48.375998 (+ 4205us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:48.376008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:48.376821 (+ 813us) server_negotiation.cc:407] Connection header received
0619 14:51:48.378608 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:48.378612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:48.378685 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:48.378787 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:48.380315 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.381051 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.382573 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.383008 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.383103 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.391006 (+ 7903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:48.391011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:48.391014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:48.391047 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:48.393668 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:48.394366 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:48.394370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:48.394372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:48.394560 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:48.394953 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:48.394955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:48.394957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:48.395204 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:48.395325 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:48.396076 (+ 751us) server_negotiation.cc:299] Negotiation successful
0619 14:51:48.396241 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4040,"thread_start_us":3923,"threads_started":1}
I20260619 14:51:48.397508 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.369751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33007 (local address 127.0.241.3:46971)
0619 14:51:48.376588 (+ 6837us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:48.376634 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:48.378359 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:48.379005 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:48.379013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:48.379524 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:48.380153 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:48.380174 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.381205 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.381209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:48.382427 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:48.382439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.383253 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.383260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:48.383584 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.388055 (+ 4471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:48.388087 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:48.390688 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:48.393792 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:48.393798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:48.393812 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:48.394255 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:48.394654 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:48.394657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:48.394660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:48.394833 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:48.395331 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:48.395345 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:48.395764 (+ 419us) client_negotiation.cc:769] Sending connection context
0619 14:51:48.396014 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 14:51:48.396404 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6552,"thread_start_us":6321,"threads_started":1}
I20260619 14:51:48.398984 12765 heartbeater.cc:347] Connected to a master server at 127.0.241.62:33007
I20260619 14:51:48.399293 12765 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:48.400142 12765 heartbeater.cc:511] Master 127.0.241.62:33007 requested a full tablet report, sending...
I20260619 14:51:48.401456 12275 ts_manager.cc:194] Registered new tserver with Master: d99dc864df914e9785066ebf63831fa7 (127.0.241.3:45687)
I20260619 14:51:48.402184 12275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:46971
I20260619 14:51:48.412101 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:51:48.428486 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.416409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.0.241.62:33007)
0619 14:51:48.416563 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:48.416567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:48.416989 (+ 422us) server_negotiation.cc:407] Connection header received
0619 14:51:48.417129 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:48.417132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:48.417204 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:48.417308 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:48.418596 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.419296 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.420665 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.421556 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.421646 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.423975 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:48.423980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:48.423983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:48.424012 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:48.426333 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:48.426912 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:48.426916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:48.426918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:48.427028 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:48.427342 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:48.427345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:48.427347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:48.427662 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:48.427856 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:48.428096 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 14:51:48.428265 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:51:48.432468 12275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:51:48.439317 12275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:51:48.467797 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.452725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:45687 (local address 127.0.0.1:42318)
0619 14:51:48.456331 (+ 3606us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:48.456353 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:48.456572 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:48.457673 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:48.457681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:48.457727 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:48.458266 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:48.458283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.460914 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.460923 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:48.464206 (+ 3283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:48.464226 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.466673 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.466680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:48.467025 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.467079 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 14:51:48.467220 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 14:51:48.467402 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3457,"thread_start_us":12369,"threads_started":1}
I20260619 14:51:48.467823 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.452807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.0.241.3:45687)
0619 14:51:48.453656 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:48.453669 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:48.456457 (+ 2788us) server_negotiation.cc:407] Connection header received
0619 14:51:48.456808 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:48.456819 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:48.457089 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:48.457270 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:48.458541 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.460667 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.464886 (+ 4219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.466449 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.466586 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.466953 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:48.467292 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 14:51:48.467477 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":628,"threads_started":1}
I20260619 14:51:48.468819 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.452310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:41981 (local address 127.0.0.1:56790)
0619 14:51:48.455364 (+ 3054us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:48.455421 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:48.455663 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:48.456814 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:48.456825 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:48.456867 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:48.458677 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:48.458693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.462941 (+ 4248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.462950 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:48.466006 (+ 3056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:48.466020 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.468054 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.468060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:48.468215 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.468247 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:51:48.468341 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:51:48.468456 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2761,"thread_start_us":3156,"threads_started":1}
I20260619 14:51:48.469571 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.452543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.0.241.1:41981)
0619 14:51:48.454225 (+ 1682us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:48.454236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:48.455569 (+ 1333us) server_negotiation.cc:407] Connection header received
0619 14:51:48.455948 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:48.455958 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:48.456218 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:48.456451 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:48.458923 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.462734 (+ 3811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.466226 (+ 3492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.467573 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.467693 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.467912 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:48.469096 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0619 14:51:48.469262 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1480,"thread_start_us":1327,"threads_started":1}
I20260619 14:51:48.472432 12700 tablet_service.cc:1467] Processing CreateTablet for tablet 55fcb0d478d54275913f4eca295ee132 (DEFAULT_TABLE table=test-table [id=238a8bf91a8f44408b7d2cbe056de2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:48.473090 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.453080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:34095 (local address 127.0.0.1:32858)
0619 14:51:48.465345 (+ 12265us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:48.465369 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:48.465568 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:48.466447 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:48.466452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:48.466479 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:48.466881 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:48.466893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.468676 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.468680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:48.470594 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:48.470608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.472333 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.472340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:48.472527 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.472559 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:51:48.472652 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:51:48.472772 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12147,"thread_start_us":2949,"threads_started":1}
I20260619 14:51:48.473140 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.453481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.0.241.2:34095)
0619 14:51:48.464759 (+ 11278us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:48.464765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:48.465471 (+ 706us) server_negotiation.cc:407] Connection header received
0619 14:51:48.465773 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:48.465781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:48.466018 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:48.466191 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:48.467050 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.468499 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.470765 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.472133 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.472280 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.472458 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:48.472701 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 14:51:48.472850 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11122,"thread_start_us":11038,"threads_started":1}
I20260619 14:51:48.474262 12700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fcb0d478d54275913f4eca295ee132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:48.474521 12423 tablet_service.cc:1467] Processing CreateTablet for tablet 55fcb0d478d54275913f4eca295ee132 (DEFAULT_TABLE table=test-table [id=238a8bf91a8f44408b7d2cbe056de2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:48.475927 12423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fcb0d478d54275913f4eca295ee132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:48.476635 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 55fcb0d478d54275913f4eca295ee132 (DEFAULT_TABLE table=test-table [id=238a8bf91a8f44408b7d2cbe056de2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:48.477938 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55fcb0d478d54275913f4eca295ee132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:48.485790 12784 tablet_bootstrap.cc:492] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7: Bootstrap starting.
I20260619 14:51:48.488832 12784 tablet_bootstrap.cc:654] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:48.490314 12784 log.cc:824] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:48.491019 12785 tablet_bootstrap.cc:492] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a: Bootstrap starting.
I20260619 14:51:48.492587 12787 tablet_bootstrap.cc:492] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171: Bootstrap starting.
I20260619 14:51:48.495112 12785 tablet_bootstrap.cc:654] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:48.495601 12787 tablet_bootstrap.cc:654] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:48.496284 12785 log.cc:824] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:48.496846 12787 log.cc:824] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:48.498071 12784 tablet_bootstrap.cc:492] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7: No bootstrap required, opened a new log
I20260619 14:51:48.498411 12784 ts_tablet_manager.cc:1397] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260619 14:51:48.499568 12785 tablet_bootstrap.cc:492] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a: No bootstrap required, opened a new log
I20260619 14:51:48.499804 12785 ts_tablet_manager.cc:1397] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 14:51:48.500154 12787 tablet_bootstrap.cc:492] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171: No bootstrap required, opened a new log
I20260619 14:51:48.500386 12787 ts_tablet_manager.cc:1397] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260619 14:51:48.507239 12785 raft_consensus.cc:348] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff531f5b2f4294ba11570565285a2a" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41981 } } peers { permanent_uuid: "d99dc864df914e9785066ebf63831fa7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45687 } } peers { permanent_uuid: "84aa514155af43e8adf5a45374abf171" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34095 } }
I20260619 14:51:48.507674 12785 raft_consensus.cc:374] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:48.507787 12785 raft_consensus.cc:729] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ff531f5b2f4294ba11570565285a2a, State: Initialized, Role: FOLLOWER
I20260619 14:51:48.508414 12785 consensus_queue.cc:260] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff531f5b2f4294ba11570565285a2a" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41981 } } peers { permanent_uuid: "d99dc864df914e9785066ebf63831fa7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45687 } } peers { permanent_uuid: "84aa514155af43e8adf5a45374abf171" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34095 } }
I20260619 14:51:48.509505 12784 raft_consensus.cc:348] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff531f5b2f4294ba11570565285a2a" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41981 } } peers { permanent_uuid: "d99dc864df914e9785066ebf63831fa7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45687 } } peers { permanent_uuid: "84aa514155af43e8adf5a45374abf171" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34095 } }
I20260619 14:51:48.509919 12784 raft_consensus.cc:374] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:48.510082 12785 ts_tablet_manager.cc:1428] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260619 14:51:48.510075 12784 raft_consensus.cc:729] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99dc864df914e9785066ebf63831fa7, State: Initialized, Role: FOLLOWER
I20260619 14:51:48.510738 12784 consensus_queue.cc:260] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff531f5b2f4294ba11570565285a2a" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41981 } } peers { permanent_uuid: "d99dc864df914e9785066ebf63831fa7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45687 } } peers { permanent_uuid: "84aa514155af43e8adf5a45374abf171" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34095 } }
I20260619 14:51:48.511276 12787 raft_consensus.cc:348] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff531f5b2f4294ba11570565285a2a" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41981 } } peers { permanent_uuid: "d99dc864df914e9785066ebf63831fa7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45687 } } peers { permanent_uuid: "84aa514155af43e8adf5a45374abf171" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34095 } }
I20260619 14:51:48.511727 12787 raft_consensus.cc:374] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:48.511844 12787 raft_consensus.cc:729] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84aa514155af43e8adf5a45374abf171, State: Initialized, Role: FOLLOWER
I20260619 14:51:48.511860 12765 heartbeater.cc:503] Master 127.0.241.62:33007 was elected leader, sending a full tablet report...
I20260619 14:51:48.512403 12787 consensus_queue.cc:260] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff531f5b2f4294ba11570565285a2a" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41981 } } peers { permanent_uuid: "d99dc864df914e9785066ebf63831fa7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45687 } } peers { permanent_uuid: "84aa514155af43e8adf5a45374abf171" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34095 } }
I20260619 14:51:48.512998 12784 ts_tablet_manager.cc:1428] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260619 14:51:48.513921 12631 heartbeater.cc:503] Master 127.0.241.62:33007 was elected leader, sending a full tablet report...
I20260619 14:51:48.514638 12787 ts_tablet_manager.cc:1428] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260619 14:51:48.530789 12489 tablet.cc:2369] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:51:48.570600 12632 tablet.cc:2369] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:51:48.624468 12766 tablet.cc:2369] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:51:48.748348 12790 raft_consensus.cc:482] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:48.748673 12790 raft_consensus.cc:504] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff531f5b2f4294ba11570565285a2a" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41981 } } peers { permanent_uuid: "d99dc864df914e9785066ebf63831fa7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45687 } } peers { permanent_uuid: "84aa514155af43e8adf5a45374abf171" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34095 } }
I20260619 14:51:48.750365 12790 leader_election.cc:288] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d99dc864df914e9785066ebf63831fa7 (127.0.241.3:45687), 84aa514155af43e8adf5a45374abf171 (127.0.241.2:34095)
I20260619 14:51:48.759336 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.750938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:43431 (local address 127.0.241.2:34095)
0619 14:51:48.751156 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:48.751163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:48.751784 (+ 621us) server_negotiation.cc:407] Connection header received
0619 14:51:48.751977 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:48.751981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:48.752051 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:48.752151 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:48.752979 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.754065 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.757010 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.758255 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.758386 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.758483 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:48.758942 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 14:51:48.759066 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 14:51:48.759323 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.750715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:45451 (local address 127.0.241.3:45687)
0619 14:51:48.750930 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:48.750936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:48.751767 (+ 831us) server_negotiation.cc:407] Connection header received
0619 14:51:48.751969 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:48.751974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:48.752051 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:48.752151 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:48.752979 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.754065 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.757007 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.758255 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.758386 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.758485 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:48.758942 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 14:51:48.759066 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:51:48.759649 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.750619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:34095 (local address 127.0.241.1:43431)
0619 14:51:48.751668 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:48.751684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:48.751808 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:48.752183 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:48.752188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:48.752222 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:48.752772 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:48.752793 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.754202 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.754207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:48.756846 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:48.756859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.758513 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.758518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:48.758739 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.758808 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 14:51:48.758907 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:51:48.759115 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":256,"spinlock_wait_cycles":13952,"thread_start_us":404,"threads_started":1}
I20260619 14:51:48.759675 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.750619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:45687 (local address 127.0.241.1:45451)
0619 14:51:48.751591 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:48.751612 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:48.751808 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:48.752183 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:48.752188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:48.752222 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:48.752831 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:48.752841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.754203 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.754207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:48.756844 (+ 2637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:48.756859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.758496 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.758503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:48.758746 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.758808 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 14:51:48.758915 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:51:48.759115 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":350,"spinlock_wait_cycles":48128,"thread_start_us":717,"threads_started":1}
I20260619 14:51:48.760656 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fcb0d478d54275913f4eca295ee132" candidate_uuid: "86ff531f5b2f4294ba11570565285a2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99dc864df914e9785066ebf63831fa7" is_pre_election: true
I20260619 14:51:48.760658 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fcb0d478d54275913f4eca295ee132" candidate_uuid: "86ff531f5b2f4294ba11570565285a2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84aa514155af43e8adf5a45374abf171" is_pre_election: true
I20260619 14:51:48.761122 12720 raft_consensus.cc:2454] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86ff531f5b2f4294ba11570565285a2a in term 0.
I20260619 14:51:48.761142 12586 raft_consensus.cc:2454] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86ff531f5b2f4294ba11570565285a2a in term 0.
I20260619 14:51:48.761807 12378 leader_election.cc:302] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 84aa514155af43e8adf5a45374abf171, 86ff531f5b2f4294ba11570565285a2a, d99dc864df914e9785066ebf63831fa7; no voters:
I20260619 14:51:48.762212 12790 raft_consensus.cc:2781] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:51:48.762363 12790 raft_consensus.cc:482] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:51:48.762478 12790 raft_consensus.cc:3037] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:48.764632 12790 raft_consensus.cc:504] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff531f5b2f4294ba11570565285a2a" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41981 } } peers { permanent_uuid: "d99dc864df914e9785066ebf63831fa7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45687 } } peers { permanent_uuid: "84aa514155af43e8adf5a45374abf171" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34095 } }
I20260619 14:51:48.765241 12790 leader_election.cc:288] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [CANDIDATE]: Term 1 election: Requested vote from peers d99dc864df914e9785066ebf63831fa7 (127.0.241.3:45687), 84aa514155af43e8adf5a45374abf171 (127.0.241.2:34095)
I20260619 14:51:48.765576 12720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fcb0d478d54275913f4eca295ee132" candidate_uuid: "86ff531f5b2f4294ba11570565285a2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d99dc864df914e9785066ebf63831fa7"
I20260619 14:51:48.765702 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55fcb0d478d54275913f4eca295ee132" candidate_uuid: "86ff531f5b2f4294ba11570565285a2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84aa514155af43e8adf5a45374abf171"
I20260619 14:51:48.765777 12720 raft_consensus.cc:3037] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:48.765895 12586 raft_consensus.cc:3037] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:48.769407 12720 raft_consensus.cc:2454] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86ff531f5b2f4294ba11570565285a2a in term 1.
I20260619 14:51:48.769424 12586 raft_consensus.cc:2454] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86ff531f5b2f4294ba11570565285a2a in term 1.
I20260619 14:51:48.769827 12376 leader_election.cc:302] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84aa514155af43e8adf5a45374abf171, 86ff531f5b2f4294ba11570565285a2a; no voters:
I20260619 14:51:48.770164 12790 raft_consensus.cc:2781] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:48.770618 12790 raft_consensus.cc:686] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 1 LEADER]: Becoming Leader. State: Replica: 86ff531f5b2f4294ba11570565285a2a, State: Running, Role: LEADER
I20260619 14:51:48.771080 12790 consensus_queue.cc:237] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff531f5b2f4294ba11570565285a2a" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41981 } } peers { permanent_uuid: "d99dc864df914e9785066ebf63831fa7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45687 } } peers { permanent_uuid: "84aa514155af43e8adf5a45374abf171" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34095 } }
I20260619 14:51:48.775903 12272 catalog_manager.cc:5495] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 86ff531f5b2f4294ba11570565285a2a (127.0.241.1). New cstate: current_term: 1 leader_uuid: "86ff531f5b2f4294ba11570565285a2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff531f5b2f4294ba11570565285a2a" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 41981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d99dc864df914e9785066ebf63831fa7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84aa514155af43e8adf5a45374abf171" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34095 } health_report { overall_health: UNKNOWN } } }
I20260619 14:51:48.831246 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.825935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.0.241.1:41981)
0619 14:51:48.826123 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:48.826129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:48.826149 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:51:48.826284 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:48.826290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:48.826387 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:48.826547 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:48.826920 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.827719 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.829158 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.829552 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.829693 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.830209 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:48.830685 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:48.830854 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:48.830911 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 14:51:48.830986 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:51:48.838960 12586 raft_consensus.cc:1261] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171 [term 1 FOLLOWER]: Refusing update from remote peer 86ff531f5b2f4294ba11570565285a2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:48.838960 12720 raft_consensus.cc:1261] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7 [term 1 FOLLOWER]: Refusing update from remote peer 86ff531f5b2f4294ba11570565285a2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:48.839888 12795 consensus_queue.cc:1035] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d99dc864df914e9785066ebf63831fa7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:48.840234 12790 consensus_queue.cc:1035] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "84aa514155af43e8adf5a45374abf171" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:48.868224 12274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:48.868494 12274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:48.872182 12274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 878bea2c186345fd9dd144a7392a1928: Sending DeleteTablet for 3 replicas of tablet 55fcb0d478d54275913f4eca295ee132
I20260619 14:51:48.873273 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fcb0d478d54275913f4eca295ee132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:48 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:32858
I20260619 14:51:48.873328 12700 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fcb0d478d54275913f4eca295ee132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:48 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:42318
I20260619 14:51:48.873989 12423 tablet_service.cc:1514] Processing DeleteTablet for tablet 55fcb0d478d54275913f4eca295ee132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:48 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:56790
I20260619 14:51:48.874219 12808 tablet_replica.cc:330] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171: stopping tablet replica
I20260619 14:51:48.874670 12808 raft_consensus.cc:2229] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:48.875262 12808 raft_consensus.cc:2258] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:48.876019 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 12355
I20260619 14:51:48.877519 12808 ts_tablet_manager.cc:1905] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:48.886991 12808 ts_tablet_manager.cc:1918] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:51:48.887336 12808 log.cc:1196] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/55fcb0d478d54275913f4eca295ee132
I20260619 14:51:48.887964 12808 ts_tablet_manager.cc:1939] T 55fcb0d478d54275913f4eca295ee132 P 84aa514155af43e8adf5a45374abf171: Deleting consensus metadata
I20260619 14:51:48.890072 12259 catalog_manager.cc:4841] TS 84aa514155af43e8adf5a45374abf171 (127.0.241.2:34095): tablet 55fcb0d478d54275913f4eca295ee132 (table test-table [id=238a8bf91a8f44408b7d2cbe056de2fd]) successfully deleted
I20260619 14:51:48.896050 12810 tablet_replica.cc:330] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a: stopping tablet replica
I20260619 14:51:48.896637 12810 raft_consensus.cc:2229] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 1 LEADER]: Raft consensus shutting down.
Jun 19 14:51:48 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:51:48.898538 12810 raft_consensus.cc:2258] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:48.898705 12809 tablet_replica.cc:330] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7: stopping tablet replica
I20260619 14:51:48.899147 12809 raft_consensus.cc:2229] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:48.899937 12809 raft_consensus.cc:2258] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:48.901085 12810 ts_tablet_manager.cc:1905] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:48.901705 12809 ts_tablet_manager.cc:1905] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:48.909082 12809 ts_tablet_manager.cc:1918] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:51:48.909140 12810 ts_tablet_manager.cc:1918] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:51:48.909236 12809 log.cc:1196] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/55fcb0d478d54275913f4eca295ee132
I20260619 14:51:48.909315 12810 log.cc:1196] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/55fcb0d478d54275913f4eca295ee132
I20260619 14:51:48.909463 12779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.877950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.0.241.1:41981)
0619 14:51:48.878124 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:48.878132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:48.888076 (+ 9944us) server_negotiation.cc:407] Connection header received
0619 14:51:48.888387 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:48.888406 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:48.888493 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:48.888576 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:48.891023 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.892133 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.893456 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.893920 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.894070 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.900753 (+ 6683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:48.900770 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:48.900786 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:48.900836 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:48.905100 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:48.906096 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:48.906109 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:48.906112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:48.906297 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:48.906757 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:48.906763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:48.906768 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:48.907218 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:48.908017 (+ 799us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:48.908610 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 14:51:48.908865 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:51:48.909710 12809 ts_tablet_manager.cc:1939] T 55fcb0d478d54275913f4eca295ee132 P d99dc864df914e9785066ebf63831fa7: Deleting consensus metadata
I20260619 14:51:48.909855 12810 ts_tablet_manager.cc:1939] T 55fcb0d478d54275913f4eca295ee132 P 86ff531f5b2f4294ba11570565285a2a: Deleting consensus metadata
I20260619 14:51:48.911314 12483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:51:48.911620 12261 catalog_manager.cc:4841] TS d99dc864df914e9785066ebf63831fa7 (127.0.241.3:45687): tablet 55fcb0d478d54275913f4eca295ee132 (table test-table [id=238a8bf91a8f44408b7d2cbe056de2fd]) successfully deleted
I20260619 14:51:48.911599 12262 catalog_manager.cc:4841] TS 86ff531f5b2f4294ba11570565285a2a (127.0.241.1:41981): tablet 55fcb0d478d54275913f4eca295ee132 (table test-table [id=238a8bf91a8f44408b7d2cbe056de2fd]) successfully deleted
I20260619 14:51:49.015413 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 12355
I20260619 14:51:49.033329 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 12494
Jun 19 14:51:49 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:51:49.048205 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:49.034109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.0.241.2:34095)
0619 14:51:49.034319 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:49.034324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:49.034341 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:51:49.034449 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:49.034452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:49.034516 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:49.034607 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:49.035809 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:49.036508 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:49.037817 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:49.038230 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:49.038327 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:49.042600 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:49.042614 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:49.042629 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:49.042660 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:49.045440 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:49.046170 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:49.046177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:49.046180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:49.046314 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:49.046686 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:49.046693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:49.046697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:49.047058 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:49.047368 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:49.047644 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 14:51:49.047855 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:51:49.048904 12626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:51:49.137454 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 12494
I20260619 14:51:49.154179 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 12637
Jun 19 14:51:49 dist-test-slave-p3ml krb5kdc[11799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:51:49.168958 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:49.155070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.0.241.3:45687)
0619 14:51:49.155229 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:49.155233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:49.155248 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:51:49.155300 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:49.155303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:49.155359 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:49.155514 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:49.156965 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:49.157666 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:49.158899 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:49.159260 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:49.159346 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:49.163543 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:49.163557 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:49.163572 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:49.163602 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:49.166237 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:49.166904 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:49.166912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:49.166914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:49.167029 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:49.167385 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:49.167388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:49.167390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:49.167803 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:49.168036 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:49.168417 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 14:51:49.168630 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:51:49.169565 12760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:51:49.259399 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 12637
I20260619 14:51:49.275347 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 12243
I20260619 14:51:49.276585 12305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:51:49.389683 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 12243
I20260619 14:51:49.411620 964 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:51:49.756637 964 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:51:49.756903 964 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:51:50.102787 964 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:51:50.103 UTC [11829] LOG: received smart shutdown request
2026-06-19 14:51:50.107 UTC [11829] LOG: background worker "logical replication launcher" (PID 11838) exited with exit code 1
2026-06-19 14:51:50.107 UTC [11833] LOG: shutting down
2026-06-19 14:51:50.253 UTC [11829] LOG: database system is shut down
2026-06-19T14:51:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47989 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:50 dist-test-slave-p3ml krb5kdc[12824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:50 dist-test-slave-p3ml krb5kdc[12824](info): set up 2 sockets
Jun 19 14:51:50 dist-test-slave-p3ml krb5kdc[12824](info): commencing operation
krb5kdc: starting...
W20260619 14:51:51.483764 964 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:51:51 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.241.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.241.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/rangeradmin_127.0.241.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/rangeradmin_127.0.241.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.241.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.241.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/rangerlookup_127.0.241.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/rangerlookup_127.0.241.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.241.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.241.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/HTTP_127.0.241.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/HTTP_127.0.241.20@KRBTEST.COM.keytab.
I20260619 14:51:51.800175 964 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskydHOLK/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/postgres -l logfile start
2026-06-19 14:51:54.166 UTC [12854] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:51:54.166 UTC [12854] LOG: listening on IPv4 address "127.0.241.20", port 45415
2026-06-19 14:51:54.169 UTC [12854] LOG: listening on Unix socket "/tmp/.s.PGSQL.45415"
2026-06-19 14:51:54.191 UTC [12857] LOG: database system was shut down at 2026-06-19 14:51:52 UTC
2026-06-19 14:51:54.195 UTC [12854] LOG: database system is ready to accept connections
I20260619 14:51:55.171489 964 mini_postgres.cc:96] Postgres bound to 45415
2026-06-19 14:51:55.188 UTC [12865] FATAL: database "slave" does not exist
127.0.241.20:45415 - accepting connections
I20260619 14:51:55.189184 964 mini_ranger.cc:161] Starting Ranger...
I20260619 14:51:55.214926 964 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:51:55.673069 964 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:51:55.673184 964 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-admin
2026-06-19 14:51:55,967 [I] DB FLAVOR :POSTGRES
2026-06-19 14:51:55,968 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:51:55,968 [I] Checking connection..
2026-06-19 14:51:55,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:51:56,221 [I] Checking connection passed.
2026-06-19 14:51:56,221 [I] --------- Verifying version history table ---------
2026-06-19 14:51:56,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:51:56,474 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:51:56,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:51:56,729 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:51:56,730 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:51:56,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:51:56.969 UTC [12929] WARNING: there is no transaction in progress
2026-06-19 14:51:56,975 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:51:56,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:51:57,226 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:51:57,227 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:51:57,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:51:57,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:51:57,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p3ml.c.gcp-upstream.internal','N') ;"
2026-06-19 14:51:57,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:51:58,231 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:51:58,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:51:58,465 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:51:58,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:51:58,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:51:58,957 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:51:58,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:51:59.889 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:51:59.906 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:51:59.922 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:51:59.936 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:51:59.951 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:51:59.969 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:52:00.035 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:52:00.042 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:52:00.057 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:52:00.068 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:52:00.078 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:52:00.092 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:52:00.101 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:52:00.107 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:52:00.114 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:52:00.121 UTC [13037] WARNING: there is no transaction in progress
2026-06-19 14:52:00,729 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:52:00,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:52:00,980 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:52:00,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:52:01,240 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:52:01,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:52:01,474 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:52:01,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:52:01,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:52:01,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p3ml.c.gcp-upstream.internal';"
2026-06-19 14:52:02,232 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:52:02,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:52:02,462 [I] DB_PATCHES have already been applied
I20260619 14:52:02.468258 964 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-admin:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskydHOLK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:52:02.468353 964 mini_ranger.cc:193] Using host: 127.0.241.20
I20260619 14:52:02.483018 964 mini_ranger.cc:227] Ranger admin URL: http://127.0.241.20:54905
Jun 19, 2026 2:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:52:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:52:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:52:03.634431 964 mini_ranger.cc:160] Time spent starting Ranger: real 8.445s user 0.000s sys 0.039s
Jun 19, 2026 2:52:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:52:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:52:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:52:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:52:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:52:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:52:25.351686 964 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.241.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.241.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/rangerkms_127.0.241.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.241.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/rangerkms_127.0.241.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.241.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.241.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.241.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/HTTP_127.0.241.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.241.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/HTTP_127.0.241.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:52:25 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880745, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:52:25.521309 964 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:52:25.544167 964 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:52:26.394353 964 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:52:26.394475 964 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-kms
I20260619 14:52:26.394531 964 mini_ranger_kms.cc:227] Using postgres at 127.0.241.20:45415
2026-06-19 14:52:26,782 [I] DB FLAVOR :POSTGRES
2026-06-19 14:52:26,783 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:52:26,783 [I] Checking connection
2026-06-19 14:52:26,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:52:27,031 [I] connection success
2026-06-19 14:52:27,032 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:52:27,032 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:52:27,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:52:27,293 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:52:27,293 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:52:27,293 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:52:27,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.241.20:45415/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:52:27,586 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:52:28.043360 964 mini_ranger_kms.cc:310] Created kms service
I20260619 14:52:28.219103 964 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:52:28.266593 964 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:52:28 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880745, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:52:28.667773 964 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:52:28.668181 964 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-kms:/tmp/dist-test-taskydHOLK/build/asan/bin/postgresql.jar:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskydHOLK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:52:28.668253 964 mini_ranger_kms.cc:240] Using host: 127.0.241.20
I20260619 14:52:28.682964 964 mini_ranger_kms.cc:276] Ranger KMS PID: 13217
I20260619 14:52:28.683127 964 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.241.20:38133
Jun 19, 2026 2:52:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:52:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:52:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:52:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:52:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:52:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 14:52:30 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880750, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.241.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:30.926419 964 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.405s user 0.099s sys 0.301s
I20260619 14:52:30.926669 964 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:52:30.926867 964 mini_ranger_kms.cc:401] 127.0.241.20:38133/kms/v1/keys
Jun 19 14:52:31 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880750, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.241.20@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
Jun 19 14:52:31 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880750, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.241.20@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
Jun 19 14:52:33 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:52:33.925026 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:36203
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:36203
--ranger_config_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.241.20:38133/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:34.260211 13268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:34.260494 13268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:34.260691 13268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:34.268580 13268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:34.268690 13268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:34.268795 13268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:34.268893 13268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:34.268975 13268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:34.279551 13268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.241.20:38133/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:36203
--ranger_config_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:36203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:34.280805 13268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:34.282146 13268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:34.294082 13273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:34.294137 13274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:34.294071 13276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:34.296759 13268 server_base.cc:1034] running on GCE node
I20260619 14:52:34.297552 13268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:34.298961 13268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:34.300208 13268 hybrid_clock.cc:648] HybridClock initialized: now 1781880754300175 us; error 69 us; skew 500 ppm
I20260619 14:52:34.300550 13268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:34 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:34.303614 13268 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:52:34.305053 13268 webserver.cc:460] Webserver started at http://127.0.241.62:43071/ using document root <none> and password file <none>
I20260619 14:52:34.305740 13268 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:34.305831 13268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:34.306072 13268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:52:34 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:52:34.362720 13268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "e79d9f7501bf44f39da80329e20ad287"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-p3ml"
server_key: "9c8ac36018932e9752729e11ba7e5f11"
server_key_iv: "7c43c748344422167faceafab9043c37"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:34.363708 13268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "e79d9f7501bf44f39da80329e20ad287"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-p3ml"
server_key: "9c8ac36018932e9752729e11ba7e5f11"
server_key_iv: "7c43c748344422167faceafab9043c37"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:34.369264 13268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:52:34.390384 13283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:34.392012 13268 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:52:34.392207 13268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "e79d9f7501bf44f39da80329e20ad287"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-p3ml"
server_key: "9c8ac36018932e9752729e11ba7e5f11"
server_key_iv: "7c43c748344422167faceafab9043c37"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:34.392463 13268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:34.416658 13268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:34.420876 13268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:34.421195 13268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:34.439441 13268 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:36203
I20260619 14:52:34.439528 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:36203 every 8 connection(s)
I20260619 14:52:34.440850 13268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:34.444514 13335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:34.448177 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 13268
I20260619 14:52:34.448393 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:52:34.449235 964 external_mini_cluster.cc:1428] Setting key b6a0e94a32b904bd7858b43b9054753b
I20260619 14:52:34.457589 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287: Bootstrap starting.
I20260619 14:52:34.461352 13335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:34.462626 13335 log.cc:824] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:52:34 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:34.465670 13335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287: No bootstrap required, opened a new log
I20260619 14:52:34.472170 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:34.451209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.0.241.62:36203)
0619 14:52:34.452043 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:34.452055 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:34.452093 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:52:34.453639 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:34.453659 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:34.454034 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:34.454362 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:34.455963 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.457287 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.459078 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.459822 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.459954 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:34.464750 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:34.464758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:34.464771 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:34.464802 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:34.468069 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:34.468864 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:34.468870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:34.468876 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:34.469033 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:34.469435 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:34.469438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:34.469440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:34.469947 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:34.470218 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:34.470647 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 14:52:34.471025 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":428,"threads_started":1}
I20260619 14:52:34.474730 13335 raft_consensus.cc:348] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79d9f7501bf44f39da80329e20ad287" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36203 } }
I20260619 14:52:34.475193 13335 raft_consensus.cc:374] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:34.475317 13335 raft_consensus.cc:729] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79d9f7501bf44f39da80329e20ad287, State: Initialized, Role: FOLLOWER
I20260619 14:52:34.475951 13335 consensus_queue.cc:260] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79d9f7501bf44f39da80329e20ad287" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36203 } }
I20260619 14:52:34.476184 13335 raft_consensus.cc:388] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:34.476328 13335 raft_consensus.cc:482] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:34.476500 13335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:34.478868 13335 raft_consensus.cc:504] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79d9f7501bf44f39da80329e20ad287" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36203 } }
I20260619 14:52:34.479358 13335 leader_election.cc:302] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e79d9f7501bf44f39da80329e20ad287; no voters:
I20260619 14:52:34.479931 13335 leader_election.cc:288] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:34.480085 13340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:34.480742 13340 raft_consensus.cc:686] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 1 LEADER]: Becoming Leader. State: Replica: e79d9f7501bf44f39da80329e20ad287, State: Running, Role: LEADER
I20260619 14:52:34.481295 13340 consensus_queue.cc:237] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79d9f7501bf44f39da80329e20ad287" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36203 } }
I20260619 14:52:34.481904 13335 sys_catalog.cc:564] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:34.484122 13341 sys_catalog.cc:455] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e79d9f7501bf44f39da80329e20ad287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79d9f7501bf44f39da80329e20ad287" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36203 } } }
I20260619 14:52:34.484573 13341 sys_catalog.cc:458] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:34.485426 13342 sys_catalog.cc:455] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e79d9f7501bf44f39da80329e20ad287. Latest consensus state: current_term: 1 leader_uuid: "e79d9f7501bf44f39da80329e20ad287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79d9f7501bf44f39da80329e20ad287" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36203 } } }
I20260619 14:52:34.485706 13342 sys_catalog.cc:458] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:34.486936 13347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:34.492158 13347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:34.493811 13335 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:52:34.509496 13347 catalog_manager.cc:1347] Generated new cluster ID: 3398fa2a21a8495aab12c95906afd48a
I20260619 14:52:34.509644 13347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:34.539718 13347 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:34.541749 13347 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:34.556128 13347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287: Generated new TSK 0
I20260619 14:52:34.557253 13347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:52:35.252952 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.241.20:38133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:36203
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:52:35 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 14:52:35.650697 13380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:35.651093 13380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:35.651304 13380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:35.663203 13380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:35.663375 13380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:35.663617 13380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:52:35.679630 13380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.241.20:38133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:35.681797 13380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:35.683694 13380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:35.704382 13390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:35.705818 13391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:35.714072 13380 server_base.cc:1034] running on GCE node
W20260619 14:52:35.714740 13393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:35.715726 13380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:35.717144 13380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:35.718452 13380 hybrid_clock.cc:648] HybridClock initialized: now 1781880755718344 us; error 94 us; skew 500 ppm
I20260619 14:52:35.718883 13380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:35 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:35.723496 13380 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:52:35.734618 13380 webserver.cc:460] Webserver started at http://127.0.241.1:33795/ using document root <none> and password file <none>
I20260619 14:52:35.735762 13380 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:35.735963 13380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:35.736393 13380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:52:35 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:52:35.768505 13380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "afaea075b0334bf3b1be3d55059e65bf"
format_stamp: "Formatted at 2026-06-19 14:52:35 on dist-test-slave-p3ml"
server_key: "0440924faf6dfbf73206313d41a85eb1"
server_key_iv: "13a5d32b17c9867ea31721c99f65e7bd"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:35.769761 13380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "afaea075b0334bf3b1be3d55059e65bf"
format_stamp: "Formatted at 2026-06-19 14:52:35 on dist-test-slave-p3ml"
server_key: "0440924faf6dfbf73206313d41a85eb1"
server_key_iv: "13a5d32b17c9867ea31721c99f65e7bd"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:35.788514 13380 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.006s sys 0.007s
I20260619 14:52:35.818789 13400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:35.820905 13380 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.009s sys 0.000s
I20260619 14:52:35.821225 13380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "afaea075b0334bf3b1be3d55059e65bf"
format_stamp: "Formatted at 2026-06-19 14:52:35 on dist-test-slave-p3ml"
server_key: "0440924faf6dfbf73206313d41a85eb1"
server_key_iv: "13a5d32b17c9867ea31721c99f65e7bd"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:35.821636 13380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:35.858520 13380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:35.868640 13380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:35.869029 13380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:35.870566 13380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:35.872607 13380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:35.872730 13380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:35.872943 13380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:35.873014 13380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:52:36 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:52:36.304940 13512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:37175 every 8 connection(s)
I20260619 14:52:36.305344 13380 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:37175
I20260619 14:52:36.307267 13380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:36.314073 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 13380
I20260619 14:52:36.314272 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:52:36.315088 964 external_mini_cluster.cc:1428] Setting key 2e6ab8658547d1dd182c1b176b82749b
Jun 19 14:52:36 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
I20260619 14:52:36.381139 13514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:36.315759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:54825 (local address 127.0.241.62:36203)
0619 14:52:36.341942 (+ 26183us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:36.341951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:36.344098 (+ 2147us) server_negotiation.cc:407] Connection header received
0619 14:52:36.346688 (+ 2590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:36.346694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:36.346808 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:36.346931 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:36.361406 (+ 14475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:36.362541 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:36.365462 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:36.366355 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:36.366488 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:36.371924 (+ 5436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:36.371935 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:36.371944 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:36.372010 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:36.376602 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:36.377762 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:36.377769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:36.377774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:36.377963 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:36.378622 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:36.378628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:36.378632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:36.379073 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:36.379237 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:36.380264 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0619 14:52:36.380512 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26030,"thread_start_us":31398,"threads_started":1}
I20260619 14:52:36.382161 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:36.315311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:36203 (local address 127.0.241.1:54825)
0619 14:52:36.342514 (+ 27203us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:36.342594 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:36.346199 (+ 3605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:36.358088 (+ 11889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:36.358103 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:36.358816 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:36.359788 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:36.359821 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:36.362809 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:36.362818 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:36.365254 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:36.365276 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:36.366914 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:36.366924 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:36.367277 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:36.368277 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:36.368320 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:36.371613 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:36.376845 (+ 5232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:36.376856 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:36.376878 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:36.377583 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:36.378153 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:36.378162 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:36.378167 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:36.378450 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:36.379354 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:36.379364 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:36.379799 (+ 435us) client_negotiation.cc:769] Sending connection context
0619 14:52:36.380115 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 14:52:36.380695 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26869,"thread_start_us":27127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
I20260619 14:52:36.385542 13513 heartbeater.cc:347] Connected to a master server at 127.0.241.62:36203
I20260619 14:52:36.385943 13513 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:36.386924 13513 heartbeater.cc:511] Master 127.0.241.62:36203 requested a full tablet report, sending...
I20260619 14:52:36.389920 13300 ts_manager.cc:194] Registered new tserver with Master: afaea075b0334bf3b1be3d55059e65bf (127.0.241.1:37175)
I20260619 14:52:36.392205 13300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:54825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:52:36.417011 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.241.20:38133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:36203
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:52:36 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
W20260619 14:52:36.915998 13519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:36.916617 13519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:36.916831 13519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:36.928411 13519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:36.928643 13519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:36.928887 13519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:52:36.944619 13519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.241.20:38133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:36.946432 13519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:36.948225 13519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:36.964768 13533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:36.971940 13534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:36.973359 13519 server_base.cc:1034] running on GCE node
W20260619 14:52:36.974231 13537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:36.975049 13519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:36.976256 13519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:36.977524 13519 hybrid_clock.cc:648] HybridClock initialized: now 1781880756977460 us; error 57 us; skew 500 ppm
I20260619 14:52:36.977910 13519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:36 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:36.981551 13519 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:52:36.983645 13519 webserver.cc:460] Webserver started at http://127.0.241.2:42883/ using document root <none> and password file <none>
I20260619 14:52:36.984426 13519 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:36.984583 13519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:36.984943 13519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:52:36 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:52:37.004606 13519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "c08773bde344451995c31af5ec3bef93"
format_stamp: "Formatted at 2026-06-19 14:52:36 on dist-test-slave-p3ml"
server_key: "73e93bdc2fb055ba7758504ecf452463"
server_key_iv: "7bfbf84beee80daed635c9bed8167669"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:37.006907 13519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "c08773bde344451995c31af5ec3bef93"
format_stamp: "Formatted at 2026-06-19 14:52:36 on dist-test-slave-p3ml"
server_key: "73e93bdc2fb055ba7758504ecf452463"
server_key_iv: "7bfbf84beee80daed635c9bed8167669"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:37.013212 13519 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260619 14:52:37.031805 13543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:37.033527 13519 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:52:37.033773 13519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "c08773bde344451995c31af5ec3bef93"
format_stamp: "Formatted at 2026-06-19 14:52:36 on dist-test-slave-p3ml"
server_key: "73e93bdc2fb055ba7758504ecf452463"
server_key_iv: "7bfbf84beee80daed635c9bed8167669"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:37.034035 13519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:37.069135 13519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:37.073688 13519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:37.074041 13519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:37.075229 13519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:37.076890 13519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:37.077019 13519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:37.077191 13519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:37.077270 13519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:37.124989 13519 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:42765
I20260619 14:52:37.125008 13655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:42765 every 8 connection(s)
I20260619 14:52:37.126332 13519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:37.130357 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 13519
I20260619 14:52:37.130558 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:52:37.131274 964 external_mini_cluster.cc:1428] Setting key 59c311f6059a7f905d727a64e56f0e49
Jun 19 14:52:37 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
I20260619 14:52:37.166679 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.136003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:34247 (local address 127.0.241.62:36203)
0619 14:52:37.143925 (+ 7922us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.143934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.145087 (+ 1153us) server_negotiation.cc:407] Connection header received
0619 14:52:37.147708 (+ 2621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.147715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.147810 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.147935 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:37.150024 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.151180 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.153929 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.154550 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.154694 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.159302 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:37.159309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:37.159314 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:37.159354 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:37.162686 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.163670 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.163675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.163679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.163832 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.164370 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.164377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.164382 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.164764 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:37.164906 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.165892 (+ 986us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.166189 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7775,"thread_start_us":7647,"threads_started":1}
I20260619 14:52:37.167826 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.135385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:36203 (local address 127.0.241.2:34247)
0619 14:52:37.144550 (+ 9165us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:37.144601 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:37.147352 (+ 2751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:37.148251 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:37.148267 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:37.148902 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:37.149812 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.149839 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.151361 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.151369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.153748 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.153775 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.154806 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.154817 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.155166 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.156049 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:37.156085 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:37.159028 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:37.162849 (+ 3821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:37.162862 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:37.162883 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:37.163531 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:37.163982 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:37.163989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:37.163996 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:37.164232 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:37.164919 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:37.164948 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:37.165406 (+ 458us) client_negotiation.cc:769] Sending connection context
0619 14:52:37.165717 (+ 311us) client_negotiation.cc:240] Negotiation successful
0619 14:52:37.166312 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8836,"thread_start_us":8570,"threads_started":1}
I20260619 14:52:37.169615 13656 heartbeater.cc:347] Connected to a master server at 127.0.241.62:36203
I20260619 14:52:37.169965 13656 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:37.170807 13656 heartbeater.cc:511] Master 127.0.241.62:36203 requested a full tablet report, sending...
I20260619 14:52:37.172299 13300 ts_manager.cc:194] Registered new tserver with Master: c08773bde344451995c31af5ec3bef93 (127.0.241.2:42765)
I20260619 14:52:37.173166 13300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:34247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:52:37.194474 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.241.20:38133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:36203
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:52:37.396096 13513 heartbeater.cc:503] Master 127.0.241.62:36203 was elected leader, sending a full tablet report...
W20260619 14:52:37.551391 13662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:37.551802 13662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:37.551977 13662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:37.559373 13662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:37.559564 13662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:37.559751 13662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:52:37.572626 13662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.241.20:38133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:37.574244 13662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:37.575654 13662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:37.588989 13667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:37.588949 13670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:37.589035 13668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:37.589740 13662 server_base.cc:1034] running on GCE node
I20260619 14:52:37.590340 13662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:37.591248 13662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:37.592476 13662 hybrid_clock.cc:648] HybridClock initialized: now 1781880757592435 us; error 35 us; skew 500 ppm
I20260619 14:52:37.592803 13662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:37 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:37.595781 13662 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:52:37.597298 13662 webserver.cc:460] Webserver started at http://127.0.241.3:43705/ using document root <none> and password file <none>
I20260619 14:52:37.597947 13662 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:37.598096 13662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:37.598416 13662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:52:37 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:52:37.613253 13662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "90b0c5fc5e44473da63377a601017ed8"
format_stamp: "Formatted at 2026-06-19 14:52:37 on dist-test-slave-p3ml"
server_key: "b9a1b1f0686fc692e9ef4090feaf62d1"
server_key_iv: "85f00777c11220243691e73c229fb906"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:37.614032 13662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "90b0c5fc5e44473da63377a601017ed8"
format_stamp: "Formatted at 2026-06-19 14:52:37 on dist-test-slave-p3ml"
server_key: "b9a1b1f0686fc692e9ef4090feaf62d1"
server_key_iv: "85f00777c11220243691e73c229fb906"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:37.619254 13662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 14:52:37.632866 13677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:37.634562 13662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:52:37.634724 13662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "90b0c5fc5e44473da63377a601017ed8"
format_stamp: "Formatted at 2026-06-19 14:52:37 on dist-test-slave-p3ml"
server_key: "b9a1b1f0686fc692e9ef4090feaf62d1"
server_key_iv: "85f00777c11220243691e73c229fb906"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:37.634960 13662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:37.653779 13662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:37.657734 13662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:37.658031 13662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:37.658957 13662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:37.660291 13662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:37.660360 13662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:37.660506 13662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:37.660564 13662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:37.694902 13662 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:38775
I20260619 14:52:37.694934 13789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:38775 every 8 connection(s)
I20260619 14:52:37.696357 13662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:37.700536 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 13662
I20260619 14:52:37.700726 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:52:37.701159 964 external_mini_cluster.cc:1428] Setting key 938b9bda4245ecb8c3c56abad48548fb
Jun 19 14:52:37 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:37.716965 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.698924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:49039 (local address 127.0.241.62:36203)
0619 14:52:37.699518 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.699523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.700284 (+ 761us) server_negotiation.cc:407] Connection header received
0619 14:52:37.702473 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.702479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.702575 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.702703 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:37.704229 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.704996 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.706773 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.707246 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.707399 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.710791 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:37.710797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:37.710800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:37.710841 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:37.713585 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.714327 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.714338 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.714341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.714467 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.715011 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.715017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.715025 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.715380 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:37.715594 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.716441 (+ 847us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.716700 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":341,"threads_started":1}
I20260619 14:52:37.717882 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.699371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:36203 (local address 127.0.241.3:49039)
0619 14:52:37.700108 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:37.700154 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:37.702193 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:37.702947 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:37.702955 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:37.703383 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:37.704050 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.704068 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.705160 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.705166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.706596 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.706616 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.707512 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.707519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.707774 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.708388 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:37.708412 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:37.710552 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:37.713736 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:37.713747 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:37.713762 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:37.714189 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:37.714587 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:37.714594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:37.714598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:37.714843 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:37.715557 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:37.715563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:37.715976 (+ 413us) client_negotiation.cc:769] Sending connection context
0619 14:52:37.716318 (+ 342us) client_negotiation.cc:240] Negotiation successful
0619 14:52:37.716760 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":284,"threads_started":1}
I20260619 14:52:37.719208 13790 heartbeater.cc:347] Connected to a master server at 127.0.241.62:36203
I20260619 14:52:37.719570 13790 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:37.720235 13790 heartbeater.cc:511] Master 127.0.241.62:36203 requested a full tablet report, sending...
I20260619 14:52:37.721701 13300 ts_manager.cc:194] Registered new tserver with Master: 90b0c5fc5e44473da63377a601017ed8 (127.0.241.3:38775)
I20260619 14:52:37.722489 13300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:49039
I20260619 14:52:37.728698 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:52:37.730283 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 13380
Jun 19 14:52:37 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:52:37.746601 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.731139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.0.241.1:37175)
0619 14:52:37.731783 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.731789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.731813 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:52:37.731865 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.731872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.732040 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.732159 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:37.733467 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.734471 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.736005 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.736548 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.736643 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.741059 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:37.741070 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:37.741084 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:37.741120 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:37.743753 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.744428 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.744434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.744436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.744571 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.744953 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.744963 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.744966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.745289 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:37.745642 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.745987 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.746248 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":364,"threads_started":1}
I20260619 14:52:37.747843 13508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:37.830978 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 13380
I20260619 14:52:37.849262 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 13519
Jun 19 14:52:37 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:52:37.864446 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.850067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.0.241.2:42765)
0619 14:52:37.850586 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.850592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.850618 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:52:37.850676 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.850683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.850893 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.851018 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:37.852229 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.853199 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.854481 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.855015 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.855105 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.859296 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:37.859306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:37.859319 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:37.859354 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:37.862134 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.862711 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.862716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.862719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.862854 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.863150 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.863152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.863154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.863513 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:37.863745 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.863982 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.864165 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":271,"threads_started":1}
I20260619 14:52:37.865638 13651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:37.947250 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 13519
I20260619 14:52:37.964977 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 13662
Jun 19 14:52:37 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:52:37.980445 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.966026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.0.241.3:38775)
0619 14:52:37.966642 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.966650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.966677 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:52:37.966748 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.966754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.966970 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.967092 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:37.968445 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.969403 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.970561 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.971084 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.971174 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.975242 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:37.975252 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:37.975265 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:37.975294 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:37.978053 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.978628 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.978634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.978636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.978762 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.979072 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.979075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.979077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.979371 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:37.979668 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.979851 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.980072 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":344,"threads_started":1}
I20260619 14:52:37.981791 13785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:38.063934 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 13662
I20260619 14:52:38.079107 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 13268
I20260619 14:52:38.080502 13330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:38.185752 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 13268
I20260619 14:52:38.207307 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:36203
--webserver_interface=127.0.241.62
--webserver_port=43071
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:36203
--ranger_config_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.241.20:38133/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.241.20:38133/kms with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:38.525734 13802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:38.526054 13802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:38.526247 13802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:38.533332 13802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:38.533489 13802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:38.533598 13802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:38.533700 13802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:38.533772 13802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:38.544020 13802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.241.20:38133/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:36203
--ranger_config_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:36203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=43071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:38.545379 13802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:38.546713 13802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:38.558149 13807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:38.558249 13808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:38.558216 13810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:38.559187 13802 server_base.cc:1034] running on GCE node
I20260619 14:52:38.559970 13802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:38.561328 13802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:38.562558 13802 hybrid_clock.cc:648] HybridClock initialized: now 1781880758562514 us; error 58 us; skew 500 ppm
I20260619 14:52:38.562902 13802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:38 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:38.566033 13802 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:52:38.567600 13802 webserver.cc:460] Webserver started at http://127.0.241.62:43071/ using document root <none> and password file <none>
I20260619 14:52:38.568213 13802 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:38.568322 13802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:52:38 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:52:38.653754 13802 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:52:38.656536 13818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:38.657845 13802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:52:38.658037 13802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "e79d9f7501bf44f39da80329e20ad287"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-p3ml"
server_key: "9c8ac36018932e9752729e11ba7e5f11"
server_key_iv: "7c43c748344422167faceafab9043c37"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:38.658967 13802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:38.675258 13802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:38.680677 13802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:38.681001 13802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:38.699049 13802 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:36203
I20260619 14:52:38.699062 13869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:36203 every 8 connection(s)
I20260619 14:52:38.700542 13802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:38.707072 13870 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:52:38.710295 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287: Bootstrap starting.
I20260619 14:52:38.710862 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 13802
I20260619 14:52:38.711910 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.241.20:38133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:37175
--local_ip_for_outbound_sockets=127.0.241.1
--tserver_master_addrs=127.0.241.62:36203
--webserver_port=33795
--webserver_interface=127.0.241.1
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.241.20:38133/kms with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:52:38.731683 13870 log.cc:824] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:38.740515 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:52:38.741081 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287: Bootstrap complete.
I20260619 14:52:38.750170 13870 raft_consensus.cc:348] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79d9f7501bf44f39da80329e20ad287" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36203 } }
I20260619 14:52:38.750995 13870 raft_consensus.cc:729] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79d9f7501bf44f39da80329e20ad287, State: Initialized, Role: FOLLOWER
I20260619 14:52:38.751657 13870 consensus_queue.cc:260] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79d9f7501bf44f39da80329e20ad287" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36203 } }
I20260619 14:52:38.751909 13870 raft_consensus.cc:388] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:38.752056 13870 raft_consensus.cc:482] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:38.752287 13870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:52:38.756153 13870 raft_consensus.cc:504] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79d9f7501bf44f39da80329e20ad287" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36203 } }
I20260619 14:52:38.756645 13870 leader_election.cc:302] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e79d9f7501bf44f39da80329e20ad287; no voters:
I20260619 14:52:38.757328 13870 leader_election.cc:288] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:52:38.757472 13874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:52:38.758553 13874 raft_consensus.cc:686] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [term 2 LEADER]: Becoming Leader. State: Replica: e79d9f7501bf44f39da80329e20ad287, State: Running, Role: LEADER
I20260619 14:52:38.759253 13874 consensus_queue.cc:237] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79d9f7501bf44f39da80329e20ad287" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36203 } }
I20260619 14:52:38.759598 13870 sys_catalog.cc:564] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:38.762048 13876 sys_catalog.cc:455] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e79d9f7501bf44f39da80329e20ad287. Latest consensus state: current_term: 2 leader_uuid: "e79d9f7501bf44f39da80329e20ad287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79d9f7501bf44f39da80329e20ad287" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36203 } } }
I20260619 14:52:38.762671 13876 sys_catalog.cc:458] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:38.762468 13875 sys_catalog.cc:455] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e79d9f7501bf44f39da80329e20ad287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79d9f7501bf44f39da80329e20ad287" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36203 } } }
I20260619 14:52:38.762830 13875 sys_catalog.cc:458] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:38.775918 13882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:38.781466 13882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:38.782262 13870 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:52:38.783324 13882 catalog_manager.cc:1259] Loaded cluster ID: 3398fa2a21a8495aab12c95906afd48a
I20260619 14:52:38.783510 13882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:38.788163 13882 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:38.790248 13882 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287: Loaded TSK: 0
I20260619 14:52:38.792209 13882 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:52:39.151702 13872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:39.152179 13872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:39.152386 13872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:39.164459 13872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:39.164609 13872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:39.164866 13872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:52:39.180716 13872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.241.20:38133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:37175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=33795
--tserver_master_addrs=127.0.241.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:39.182350 13872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:39.184085 13872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:39.236020 13907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:39.238853 13908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:39.255729 13911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:39.256326 13872 server_base.cc:1034] running on GCE node
I20260619 14:52:39.257397 13872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:39.263590 13872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:39.267072 13872 hybrid_clock.cc:648] HybridClock initialized: now 1781880759266976 us; error 74 us; skew 500 ppm
I20260619 14:52:39.267587 13872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:39 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:39.274967 13872 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:52:39.283737 13872 webserver.cc:460] Webserver started at http://127.0.241.1:33795/ using document root <none> and password file <none>
I20260619 14:52:39.284694 13872 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:39.284857 13872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:52:39 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:52:39.324709 13872 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 14:52:39.340682 13919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:39.342836 13872 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.007s sys 0.008s
I20260619 14:52:39.343261 13872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "afaea075b0334bf3b1be3d55059e65bf"
format_stamp: "Formatted at 2026-06-19 14:52:35 on dist-test-slave-p3ml"
server_key: "0440924faf6dfbf73206313d41a85eb1"
server_key_iv: "13a5d32b17c9867ea31721c99f65e7bd"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:39.344836 13872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:39.367660 13872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:39.372920 13872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:39.373371 13872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:39.378201 13872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:39.382700 13872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:39.382855 13872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:39.382998 13872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:39.383093 13872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:39.451959 13872 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:37175
I20260619 14:52:39.452020 14031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:37175 every 8 connection(s)
I20260619 14:52:39.454210 13872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:39.456068 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 13872
I20260619 14:52:39.457111 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.241.20:38133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:42765
--local_ip_for_outbound_sockets=127.0.241.2
--tserver_master_addrs=127.0.241.62:36203
--webserver_port=42883
--webserver_interface=127.0.241.2
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.241.20:38133/kms with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:52:39 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:39.507704 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:39.459405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:36203 (local address 127.0.241.1:49071)
0619 14:52:39.466866 (+ 7461us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:39.466925 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:39.474665 (+ 7740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:39.477306 (+ 2641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:39.477319 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:39.477991 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:39.478834 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:39.478863 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:39.481725 (+ 2862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:39.481734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:39.484210 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:39.484242 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:39.486592 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:39.486601 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:39.486960 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:39.490924 (+ 3964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:39.490958 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:39.493691 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:39.500332 (+ 6641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:39.500342 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:39.500361 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:39.501000 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:39.503938 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:39.503944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:39.503948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:39.504212 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:39.505281 (+ 1069us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:39.505289 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:39.505655 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 14:52:39.505931 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 14:52:39.506451 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7120,"thread_start_us":6832,"threads_started":1}
I20260619 14:52:39.508803 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:39.459600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:49071 (local address 127.0.241.62:36203)
0619 14:52:39.473383 (+ 13783us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:39.473405 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:39.473468 (+ 63us) server_negotiation.cc:407] Connection header received
0619 14:52:39.475614 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:39.475646 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:39.476162 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:39.476554 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:39.479083 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:39.481452 (+ 2369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:39.484632 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:39.485925 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:39.486151 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:39.494028 (+ 7877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:39.494039 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:39.494060 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:39.494112 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:39.500138 (+ 6026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:39.503521 (+ 3383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:39.503531 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:39.503541 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:39.503781 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:39.504360 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:39.504366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:39.504370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:39.505012 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:39.505199 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:39.506971 (+ 1772us) server_negotiation.cc:299] Negotiation successful
0619 14:52:39.507406 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13472,"thread_start_us":13241,"threads_started":1}
I20260619 14:52:39.513015 14032 heartbeater.cc:347] Connected to a master server at 127.0.241.62:36203
I20260619 14:52:39.513331 14032 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:39.514144 14032 heartbeater.cc:511] Master 127.0.241.62:36203 requested a full tablet report, sending...
W20260619 14:52:39.517086 14032 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:36203 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:52:39.517311 14032 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:39.517562 14032 heartbeater.cc:511] Master 127.0.241.62:36203 requested a full tablet report, sending...
I20260619 14:52:39.518846 14032 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:39.519096 14032 heartbeater.cc:511] Master 127.0.241.62:36203 requested a full tablet report, sending...
W20260619 14:52:39.520354 14032 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:52:39.884866 14036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:39.885274 14036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:39.885428 14036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:39.897171 14036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:39.897374 14036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:39.897589 14036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:52:39.913192 14036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.241.20:38133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:42765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=42883
--tserver_master_addrs=127.0.241.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:39.914959 14036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:39.916515 14036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:39.950327 14052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:39.952137 14053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:39.963245 14055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:39.964172 14036 server_base.cc:1034] running on GCE node
I20260619 14:52:39.964948 14036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:39.966312 14036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:39.967590 14036 hybrid_clock.cc:648] HybridClock initialized: now 1781880759967524 us; error 54 us; skew 500 ppm
I20260619 14:52:39.967967 14036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:39 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:39.972358 14036 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:52:39.976939 14036 webserver.cc:460] Webserver started at http://127.0.241.2:42883/ using document root <none> and password file <none>
I20260619 14:52:39.977880 14036 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:39.978026 14036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:52:39 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
Jun 19 14:52:40 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:40.021462 14036 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.000s sys 0.010s
I20260619 14:52:40.036211 14063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:40.038479 14036 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.002s sys 0.012s
I20260619 14:52:40.043968 14036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "c08773bde344451995c31af5ec3bef93"
format_stamp: "Formatted at 2026-06-19 14:52:36 on dist-test-slave-p3ml"
server_key: "73e93bdc2fb055ba7758504ecf452463"
server_key_iv: "7bfbf84beee80daed635c9bed8167669"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:40.045717 14036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:40.103355 14036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:40.108805 14036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:40.109148 14036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:40.110394 14036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:40.112213 14036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:40.112339 14036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:40.112447 14036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:40.112524 14036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:40.192431 14036 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:42765
I20260619 14:52:40.192526 14176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:42765 every 8 connection(s)
I20260619 14:52:40.194615 14036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:40.200491 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 14036
I20260619 14:52:40.201272 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.241.20:38133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:38775
--local_ip_for_outbound_sockets=127.0.241.3
--tserver_master_addrs=127.0.241.62:36203
--webserver_port=43705
--webserver_interface=127.0.241.3
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.241.20:38133/kms with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:52:40 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:40.249032 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:40.198982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:45223 (local address 127.0.241.62:36203)
0619 14:52:40.222887 (+ 23905us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:40.222893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:40.226797 (+ 3904us) server_negotiation.cc:407] Connection header received
0619 14:52:40.226879 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:40.226885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:40.226978 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:40.227085 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:40.231623 (+ 4538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:40.232801 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:40.235474 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:40.236113 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:40.236249 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:40.241237 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:40.241245 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:40.241250 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:40.241296 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:40.245022 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:40.246054 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:40.246061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:40.246065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:40.246232 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:40.246821 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:40.246826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:40.246829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:40.247220 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:40.247369 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:40.248427 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0619 14:52:40.248660 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23731,"thread_start_us":23607,"threads_started":1}
I20260619 14:52:40.252240 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:40.198397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:36203 (local address 127.0.241.2:45223)
0619 14:52:40.223582 (+ 25185us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:40.223644 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:40.226366 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:40.229082 (+ 2716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:40.229096 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:40.229826 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:40.230737 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:40.230782 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:40.233051 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:40.233057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:40.235232 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:40.235250 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:40.236678 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:40.236690 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:40.237006 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:40.238021 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:40.238052 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:40.240915 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:40.245224 (+ 4309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:40.245235 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:40.245254 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:40.245892 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:40.246391 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:40.246396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:40.246400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:40.246674 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:40.247485 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:40.247494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:40.247952 (+ 458us) client_negotiation.cc:769] Sending connection context
0619 14:52:40.249599 (+ 1647us) client_negotiation.cc:240] Negotiation successful
0619 14:52:40.250436 (+ 837us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24804,"thread_start_us":24562,"threads_started":1}
I20260619 14:52:40.254339 14177 heartbeater.cc:347] Connected to a master server at 127.0.241.62:36203
I20260619 14:52:40.254702 14177 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:40.255648 14177 heartbeater.cc:511] Master 127.0.241.62:36203 requested a full tablet report, sending...
I20260619 14:52:40.257758 13835 ts_manager.cc:194] Registered new tserver with Master: c08773bde344451995c31af5ec3bef93 (127.0.241.2:42765)
I20260619 14:52:40.259984 13835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:45223
Jun 19 14:52:40 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:52:40.523336 14032 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:40.523952 14032 heartbeater.cc:511] Master 127.0.241.62:36203 requested a full tablet report, sending...
I20260619 14:52:40.525270 13835 ts_manager.cc:194] Registered new tserver with Master: afaea075b0334bf3b1be3d55059e65bf (127.0.241.1:37175)
I20260619 14:52:40.526511 13835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:49071
W20260619 14:52:40.675479 14181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:40.675899 14181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:40.676069 14181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:40.687774 14181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:40.687935 14181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:40.688133 14181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:52:40.703727 14181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.241.20:38133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:38775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=43705
--tserver_master_addrs=127.0.241.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:40.705586 14181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:40.707441 14181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:40.724887 14187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:40.724946 14186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:40.733892 14181 server_base.cc:1034] running on GCE node
W20260619 14:52:40.734325 14189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:40.735159 14181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:40.738153 14181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:40.739419 14181 hybrid_clock.cc:648] HybridClock initialized: now 1781880760739302 us; error 101 us; skew 500 ppm
I20260619 14:52:40.739881 14181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:40 dist-test-slave-p3ml krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:40.746519 14181 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:52:40.749040 14181 webserver.cc:460] Webserver started at http://127.0.241.3:43705/ using document root <none> and password file <none>
I20260619 14:52:40.749990 14181 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:40.750245 14181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:52:40 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:52:40.781574 14181 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.000s sys 0.006s
I20260619 14:52:40.787304 14197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:40.789173 14181 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260619 14:52:40.789533 14181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "90b0c5fc5e44473da63377a601017ed8"
format_stamp: "Formatted at 2026-06-19 14:52:37 on dist-test-slave-p3ml"
server_key: "b9a1b1f0686fc692e9ef4090feaf62d1"
server_key_iv: "85f00777c11220243691e73c229fb906"
server_key_version: "kuduclusterkey@0"
I20260619 14:52:40.791024 14181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:40.822710 14181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Jun 19 14:52:40 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for HTTP/127.0.241.20@KRBTEST.COM
I20260619 14:52:40.827796 14181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:40.828105 14181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:40.829352 14181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:40.831192 14181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:40.831302 14181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:40.831415 14181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:40.831525 14181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 14:52:41.051112 14181 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:38775
I20260619 14:52:41.051112 14310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:38775 every 8 connection(s)
I20260619 14:52:41.053099 14181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:41.054183 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 14181
Jun 19 14:52:41 dist-test-slave-p3ml krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:41.122434 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.055374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58180 (local address 127.0.241.62:36203)
0619 14:52:41.090492 (+ 35118us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:41.090503 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:41.090543 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:52:41.090662 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:41.090671 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:41.090793 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:41.090998 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:41.096731 (+ 5733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.098434 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.100532 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.101523 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.101724 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.105550 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:41.105561 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:41.105570 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:41.105634 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:41.116530 (+ 10896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:41.118489 (+ 1959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:41.118495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:41.118499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:41.118671 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:41.119174 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:41.119179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:41.119183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:41.119929 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:41.121581 (+ 1652us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:41.121707 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 14:52:41.122055 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34908,"thread_start_us":43822,"threads_started":1}
I20260619 14:52:41.126973 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.064676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:53149 (local address 127.0.241.62:36203)
0619 14:52:41.099192 (+ 34516us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:41.099202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:41.099229 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:52:41.102838 (+ 3609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:41.102844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:41.102959 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:41.103078 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:41.107635 (+ 4557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.109384 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.112176 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.113023 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.113123 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.118583 (+ 5460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:41.118589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:41.118594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:41.118637 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:41.122223 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:41.123145 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:41.123151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:41.123154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:41.123316 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:41.124684 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:41.124689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:41.124692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:41.125043 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:41.125177 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:41.126352 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0619 14:52:41.126559 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34329,"spinlock_wait_cycles":4864,"thread_start_us":26477,"threads_started":1}
I20260619 14:52:41.128201 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.063731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:36203 (local address 127.0.241.3:53149)
0619 14:52:41.092819 (+ 29088us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:41.092924 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:41.102357 (+ 9433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:41.103666 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:41.103680 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:41.104355 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:41.105294 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:41.105321 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.109599 (+ 4278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.109606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:41.112000 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:41.112015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.113548 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.113556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:41.113866 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.114666 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:41.114706 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:41.118193 (+ 3487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:41.122372 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:41.122383 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:41.122402 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:41.122993 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:41.124219 (+ 1226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:41.124224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:41.124227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:41.124543 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:41.125654 (+ 1111us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:41.125659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:41.125998 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 14:52:41.126260 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 14:52:41.126821 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28592,"thread_start_us":31582,"threads_started":1}
I20260619 14:52:41.129859 14311 heartbeater.cc:347] Connected to a master server at 127.0.241.62:36203
I20260619 14:52:41.130156 14311 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:41.130862 14311 heartbeater.cc:511] Master 127.0.241.62:36203 requested a full tablet report, sending...
I20260619 14:52:41.132346 13835 ts_manager.cc:194] Registered new tserver with Master: 90b0c5fc5e44473da63377a601017ed8 (127.0.241.3:38775)
I20260619 14:52:41.133140 13835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:53149
I20260619 14:52:41.137843 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:52:41.155947 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.141266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.0.241.62:36203)
0619 14:52:41.141467 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:41.141473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:41.141945 (+ 472us) server_negotiation.cc:407] Connection header received
0619 14:52:41.142086 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:41.142091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:41.142169 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:41.142287 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:41.143856 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.144927 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.146338 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.146969 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.147097 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.149752 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:41.149758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:41.149763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:41.149801 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:41.153320 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:41.153937 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:41.153943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:41.153947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:41.154085 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:41.154442 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:41.154460 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:41.154463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:41.154824 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:41.155072 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:41.155358 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 14:52:41.155599 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:52:41.161695 13835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:52:41.172919 13835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:52:41.227571 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.188586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.0.241.2:42765)
0619 14:52:41.217732 (+ 29146us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:41.217740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:41.218387 (+ 647us) server_negotiation.cc:407] Connection header received
0619 14:52:41.218493 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:41.218500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:41.219146 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:41.219324 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:41.220187 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.221677 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.224382 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.225760 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.225872 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.226107 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:41.227026 (+ 919us) server_negotiation.cc:299] Negotiation successful
0619 14:52:41.227213 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28968,"thread_start_us":28854,"threads_started":1}
I20260619 14:52:41.227687 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.187892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:37175 (local address 127.0.0.1:43374)
0619 14:52:41.214840 (+ 26948us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:41.214889 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:41.215086 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:41.216731 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:41.216739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:41.216770 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:41.217187 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:41.217201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.221271 (+ 4070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.221279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:41.224772 (+ 3493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:41.224785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.226903 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.226912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:41.227089 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.227126 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 14:52:41.227208 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:52:41.227336 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26635,"thread_start_us":33078,"threads_started":1}
I20260619 14:52:41.228269 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.188104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:42765 (local address 127.0.0.1:40990)
0619 14:52:41.218145 (+ 30041us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:41.218163 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:41.218275 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:41.219574 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:41.219579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:41.219603 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:41.219975 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:41.219985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.221856 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.221861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:41.224175 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:41.224189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.226263 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.226269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:41.226448 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.226495 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 14:52:41.226614 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 14:52:41.226809 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29920,"thread_start_us":29792,"threads_started":1}
I20260619 14:52:41.228652 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.188202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.0.241.1:37175)
0619 14:52:41.199344 (+ 11142us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:41.199353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:41.215850 (+ 16497us) server_negotiation.cc:407] Connection header received
0619 14:52:41.215958 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:41.215966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:41.216227 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:41.216433 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:41.218284 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.220673 (+ 2389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.224951 (+ 4278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.226401 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.226518 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.226734 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:41.228172 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0619 14:52:41.228339 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10957,"thread_start_us":11156,"threads_started":1}
I20260619 14:52:41.230301 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.198977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.0.241.3:38775)
0619 14:52:41.215740 (+ 16763us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:41.215748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:41.221365 (+ 5617us) server_negotiation.cc:407] Connection header received
0619 14:52:41.221656 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:41.221665 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:41.221863 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:41.222021 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:41.223030 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.224689 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.227043 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.228497 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.228613 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.228847 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:41.229816 (+ 969us) server_negotiation.cc:299] Negotiation successful
0619 14:52:41.229984 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16584,"thread_start_us":16398,"threads_started":1}
I20260619 14:52:41.230715 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.198437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:38775 (local address 127.0.0.1:60578)
0619 14:52:41.221222 (+ 22785us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:41.221264 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:41.221476 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:41.222355 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:41.222359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:41.222382 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:41.222825 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:41.222837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.224864 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.224868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:41.226804 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:41.226820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.229044 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.229052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:41.229229 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.229258 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:52:41.229357 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:52:41.229470 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22663,"thread_start_us":15924,"threads_started":1}
I20260619 14:52:41.232532 14112 tablet_service.cc:1467] Processing CreateTablet for tablet dd90ae0f155d4d8ca9acf23b28109765 (DEFAULT_TABLE table=test-table [id=84f513ed5a4241eb8f3202c6c171109f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:41.233152 13967 tablet_service.cc:1467] Processing CreateTablet for tablet dd90ae0f155d4d8ca9acf23b28109765 (DEFAULT_TABLE table=test-table [id=84f513ed5a4241eb8f3202c6c171109f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:41.234292 14112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd90ae0f155d4d8ca9acf23b28109765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:41.234568 13967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd90ae0f155d4d8ca9acf23b28109765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:41.234925 14246 tablet_service.cc:1467] Processing CreateTablet for tablet dd90ae0f155d4d8ca9acf23b28109765 (DEFAULT_TABLE table=test-table [id=84f513ed5a4241eb8f3202c6c171109f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:41.236225 14246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd90ae0f155d4d8ca9acf23b28109765. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:41.257076 14340 tablet_bootstrap.cc:492] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8: Bootstrap starting.
I20260619 14:52:41.257333 14339 tablet_bootstrap.cc:492] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93: Bootstrap starting.
I20260619 14:52:41.259359 14341 tablet_bootstrap.cc:492] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf: Bootstrap starting.
I20260619 14:52:41.261132 14339 tablet_bootstrap.cc:654] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:41.262486 14339 log.cc:824] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:41.262995 14340 tablet_bootstrap.cc:654] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:41.263736 14341 tablet_bootstrap.cc:654] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:41.264365 14340 log.cc:824] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:41.264999 14341 log.cc:824] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:41.265077 14177 heartbeater.cc:503] Master 127.0.241.62:36203 was elected leader, sending a full tablet report...
I20260619 14:52:41.272143 14339 tablet_bootstrap.cc:492] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93: No bootstrap required, opened a new log
I20260619 14:52:41.272475 14339 ts_tablet_manager.cc:1397] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20260619 14:52:41.278512 14340 tablet_bootstrap.cc:492] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8: No bootstrap required, opened a new log
I20260619 14:52:41.278532 14341 tablet_bootstrap.cc:492] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf: No bootstrap required, opened a new log
I20260619 14:52:41.278793 14341 ts_tablet_manager.cc:1397] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.011s
I20260619 14:52:41.278884 14340 ts_tablet_manager.cc:1397] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.013s
I20260619 14:52:41.283756 14339 raft_consensus.cc:348] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afaea075b0334bf3b1be3d55059e65bf" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37175 } } peers { permanent_uuid: "c08773bde344451995c31af5ec3bef93" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42765 } } peers { permanent_uuid: "90b0c5fc5e44473da63377a601017ed8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38775 } }
I20260619 14:52:41.284260 14339 raft_consensus.cc:374] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:41.284376 14339 raft_consensus.cc:729] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08773bde344451995c31af5ec3bef93, State: Initialized, Role: FOLLOWER
I20260619 14:52:41.285138 14339 consensus_queue.cc:260] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afaea075b0334bf3b1be3d55059e65bf" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37175 } } peers { permanent_uuid: "c08773bde344451995c31af5ec3bef93" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42765 } } peers { permanent_uuid: "90b0c5fc5e44473da63377a601017ed8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38775 } }
I20260619 14:52:41.290263 14340 raft_consensus.cc:348] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afaea075b0334bf3b1be3d55059e65bf" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37175 } } peers { permanent_uuid: "c08773bde344451995c31af5ec3bef93" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42765 } } peers { permanent_uuid: "90b0c5fc5e44473da63377a601017ed8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38775 } }
I20260619 14:52:41.290805 14340 raft_consensus.cc:374] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:41.290935 14340 raft_consensus.cc:729] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b0c5fc5e44473da63377a601017ed8, State: Initialized, Role: FOLLOWER
I20260619 14:52:41.290710 14341 raft_consensus.cc:348] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afaea075b0334bf3b1be3d55059e65bf" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37175 } } peers { permanent_uuid: "c08773bde344451995c31af5ec3bef93" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42765 } } peers { permanent_uuid: "90b0c5fc5e44473da63377a601017ed8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38775 } }
I20260619 14:52:41.291203 14341 raft_consensus.cc:374] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:41.291359 14341 raft_consensus.cc:729] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afaea075b0334bf3b1be3d55059e65bf, State: Initialized, Role: FOLLOWER
I20260619 14:52:41.291694 14340 consensus_queue.cc:260] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afaea075b0334bf3b1be3d55059e65bf" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37175 } } peers { permanent_uuid: "c08773bde344451995c31af5ec3bef93" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42765 } } peers { permanent_uuid: "90b0c5fc5e44473da63377a601017ed8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38775 } }
I20260619 14:52:41.292142 14341 consensus_queue.cc:260] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afaea075b0334bf3b1be3d55059e65bf" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37175 } } peers { permanent_uuid: "c08773bde344451995c31af5ec3bef93" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42765 } } peers { permanent_uuid: "90b0c5fc5e44473da63377a601017ed8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38775 } }
I20260619 14:52:41.295684 14339 ts_tablet_manager.cc:1428] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93: Time spent starting tablet: real 0.023s user 0.018s sys 0.001s
I20260619 14:52:41.296622 14032 heartbeater.cc:503] Master 127.0.241.62:36203 was elected leader, sending a full tablet report...
I20260619 14:52:41.297744 14341 ts_tablet_manager.cc:1428] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
I20260619 14:52:41.298660 14311 heartbeater.cc:503] Master 127.0.241.62:36203 was elected leader, sending a full tablet report...
I20260619 14:52:41.300249 14340 ts_tablet_manager.cc:1428] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8: Time spent starting tablet: real 0.021s user 0.015s sys 0.002s
W20260619 14:52:41.357769 14312 tablet.cc:2369] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:41.436686 14347 raft_consensus.cc:482] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:41.436990 14347 raft_consensus.cc:504] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afaea075b0334bf3b1be3d55059e65bf" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37175 } } peers { permanent_uuid: "c08773bde344451995c31af5ec3bef93" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42765 } } peers { permanent_uuid: "90b0c5fc5e44473da63377a601017ed8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38775 } }
I20260619 14:52:41.438889 14347 leader_election.cc:288] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c08773bde344451995c31af5ec3bef93 (127.0.241.2:42765), 90b0c5fc5e44473da63377a601017ed8 (127.0.241.3:38775)
I20260619 14:52:41.446918 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.439677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:51475 (local address 127.0.241.3:38775)
0619 14:52:41.439879 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:41.439888 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:41.440670 (+ 782us) server_negotiation.cc:407] Connection header received
0619 14:52:41.440796 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:41.440801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:41.440876 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:41.440975 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:41.441715 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.442800 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.445126 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.445866 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.446034 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.446142 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:41.446523 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 14:52:41.446659 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:52:41.446952 14348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.439227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:38775 (local address 127.0.241.1:51475)
0619 14:52:41.440557 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:41.440573 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:41.440682 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:41.441021 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:41.441029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:41.441059 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:41.441527 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:41.441546 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.442937 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.442942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:41.444985 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:41.444994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.446097 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.446103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:41.446323 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.446380 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 14:52:41.446480 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:52:41.446606 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"thread_start_us":698,"threads_started":1}
I20260619 14:52:41.447099 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.439517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:37313 (local address 127.0.241.2:42765)
0619 14:52:41.439817 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:41.439824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:41.440451 (+ 627us) server_negotiation.cc:407] Connection header received
0619 14:52:41.440747 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:41.440752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:41.440832 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:41.440881 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:41.441735 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.442431 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.444953 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.446140 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.446277 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.446391 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:41.446715 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 14:52:41.446862 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260619 14:52:41.447824 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.439161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:42765 (local address 127.0.241.1:37313)
0619 14:52:41.440320 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:41.440349 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:41.440573 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:41.441078 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:41.441085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:41.441113 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:41.441554 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:41.441563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.442574 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.442579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:41.444772 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:41.444786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.446352 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.446375 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:41.446546 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.446575 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:52:41.447293 (+ 718us) client_negotiation.cc:240] Negotiation successful
0619 14:52:41.447427 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":3,"thread_start_us":1120,"threads_started":1}
I20260619 14:52:41.448369 14266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd90ae0f155d4d8ca9acf23b28109765" candidate_uuid: "afaea075b0334bf3b1be3d55059e65bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b0c5fc5e44473da63377a601017ed8" is_pre_election: true
I20260619 14:52:41.448940 14266 raft_consensus.cc:2454] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afaea075b0334bf3b1be3d55059e65bf in term 0.
I20260619 14:52:41.448886 14132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd90ae0f155d4d8ca9acf23b28109765" candidate_uuid: "afaea075b0334bf3b1be3d55059e65bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c08773bde344451995c31af5ec3bef93" is_pre_election: true
I20260619 14:52:41.449471 14132 raft_consensus.cc:2454] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afaea075b0334bf3b1be3d55059e65bf in term 0.
I20260619 14:52:41.449566 13921 leader_election.cc:302] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b0c5fc5e44473da63377a601017ed8, afaea075b0334bf3b1be3d55059e65bf; no voters:
I20260619 14:52:41.450042 14347 raft_consensus.cc:2781] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:41.450210 14347 raft_consensus.cc:482] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:41.450325 14347 raft_consensus.cc:3037] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:41.452979 14347 raft_consensus.cc:504] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afaea075b0334bf3b1be3d55059e65bf" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37175 } } peers { permanent_uuid: "c08773bde344451995c31af5ec3bef93" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42765 } } peers { permanent_uuid: "90b0c5fc5e44473da63377a601017ed8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38775 } }
I20260619 14:52:41.453743 14347 leader_election.cc:288] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [CANDIDATE]: Term 1 election: Requested vote from peers c08773bde344451995c31af5ec3bef93 (127.0.241.2:42765), 90b0c5fc5e44473da63377a601017ed8 (127.0.241.3:38775)
I20260619 14:52:41.453961 14132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd90ae0f155d4d8ca9acf23b28109765" candidate_uuid: "afaea075b0334bf3b1be3d55059e65bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c08773bde344451995c31af5ec3bef93"
I20260619 14:52:41.454037 14266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd90ae0f155d4d8ca9acf23b28109765" candidate_uuid: "afaea075b0334bf3b1be3d55059e65bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b0c5fc5e44473da63377a601017ed8"
I20260619 14:52:41.454156 14132 raft_consensus.cc:3037] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:41.454283 14266 raft_consensus.cc:3037] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:41.456506 14132 raft_consensus.cc:2454] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afaea075b0334bf3b1be3d55059e65bf in term 1.
I20260619 14:52:41.456941 13920 leader_election.cc:302] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afaea075b0334bf3b1be3d55059e65bf, c08773bde344451995c31af5ec3bef93; no voters:
I20260619 14:52:41.457232 14347 raft_consensus.cc:2781] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:41.457309 14266 raft_consensus.cc:2454] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afaea075b0334bf3b1be3d55059e65bf in term 1.
I20260619 14:52:41.457795 14347 raft_consensus.cc:686] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 1 LEADER]: Becoming Leader. State: Replica: afaea075b0334bf3b1be3d55059e65bf, State: Running, Role: LEADER
I20260619 14:52:41.458333 14347 consensus_queue.cc:237] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afaea075b0334bf3b1be3d55059e65bf" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37175 } } peers { permanent_uuid: "c08773bde344451995c31af5ec3bef93" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42765 } } peers { permanent_uuid: "90b0c5fc5e44473da63377a601017ed8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38775 } }
W20260619 14:52:41.460026 14033 tablet.cc:2369] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:41.463265 13834 catalog_manager.cc:5495] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf reported cstate change: term changed from 0 to 1, leader changed from <none> to afaea075b0334bf3b1be3d55059e65bf (127.0.241.1). New cstate: current_term: 1 leader_uuid: "afaea075b0334bf3b1be3d55059e65bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afaea075b0334bf3b1be3d55059e65bf" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c08773bde344451995c31af5ec3bef93" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b0c5fc5e44473da63377a601017ed8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38775 } health_report { overall_health: UNKNOWN } } }
W20260619 14:52:41.482767 14178 tablet.cc:2369] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:41.498472 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.493430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.0.241.1:37175)
0619 14:52:41.493603 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:41.493607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:41.493625 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:52:41.493740 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:41.493744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:41.493832 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:41.493963 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:41.494281 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.494957 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.496503 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.496914 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.497006 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.497511 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:41.497994 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:41.498106 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:41.498198 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 14:52:41.498259 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:52:41.506436 14266 raft_consensus.cc:1261] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8 [term 1 FOLLOWER]: Refusing update from remote peer afaea075b0334bf3b1be3d55059e65bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:41.506446 14132 raft_consensus.cc:1261] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93 [term 1 FOLLOWER]: Refusing update from remote peer afaea075b0334bf3b1be3d55059e65bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:41.507513 14350 consensus_queue.cc:1035] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c08773bde344451995c31af5ec3bef93" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 42765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:41.507893 14347 consensus_queue.cc:1035] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b0c5fc5e44473da63377a601017ed8" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:41.523590 14355 mvcc.cc:205] Tried to move back new op lower bound from 7298583599120392192 to 7298583598936862720. Current Snapshot: MvccSnapshot[applied={T|T < 7298583599120392192}]
I20260619 14:52:41.532510 13834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:41.532790 13834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:41.536085 13834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e79d9f7501bf44f39da80329e20ad287: Sending DeleteTablet for 3 replicas of tablet dd90ae0f155d4d8ca9acf23b28109765
I20260619 14:52:41.537166 13967 tablet_service.cc:1514] Processing DeleteTablet for tablet dd90ae0f155d4d8ca9acf23b28109765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:41 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:43374
I20260619 14:52:41.537160 14112 tablet_service.cc:1514] Processing DeleteTablet for tablet dd90ae0f155d4d8ca9acf23b28109765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:41 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:40990
I20260619 14:52:41.537245 14246 tablet_service.cc:1514] Processing DeleteTablet for tablet dd90ae0f155d4d8ca9acf23b28109765 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:41 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:60578
I20260619 14:52:41.542099 14365 tablet_replica.cc:330] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8: stopping tablet replica
I20260619 14:52:41.542646 14365 raft_consensus.cc:2229] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:41.544093 14363 tablet_replica.cc:330] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93: stopping tablet replica
I20260619 14:52:41.544548 14363 raft_consensus.cc:2229] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:41.545125 14363 raft_consensus.cc:2258] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:41.545600 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 13872
I20260619 14:52:41.545905 14364 tablet_replica.cc:330] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf: stopping tablet replica
I20260619 14:52:41.546288 14364 raft_consensus.cc:2229] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:52:41.546949 14364 raft_consensus.cc:2258] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:41.548014 14363 ts_tablet_manager.cc:1905] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:41.548950 14364 ts_tablet_manager.cc:1905] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:41.550464 14365 raft_consensus.cc:2258] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:41.554107 14363 ts_tablet_manager.cc:1918] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:41.554302 14363 log.cc:1196] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/dd90ae0f155d4d8ca9acf23b28109765
I20260619 14:52:41.554888 14363 ts_tablet_manager.cc:1939] T dd90ae0f155d4d8ca9acf23b28109765 P c08773bde344451995c31af5ec3bef93: Deleting consensus metadata
I20260619 14:52:41.556164 14364 ts_tablet_manager.cc:1918] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:41.556365 14364 log.cc:1196] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/dd90ae0f155d4d8ca9acf23b28109765
I20260619 14:52:41.556895 14364 ts_tablet_manager.cc:1939] T dd90ae0f155d4d8ca9acf23b28109765 P afaea075b0334bf3b1be3d55059e65bf: Deleting consensus metadata
I20260619 14:52:41.556931 13819 catalog_manager.cc:4841] TS c08773bde344451995c31af5ec3bef93 (127.0.241.2:42765): tablet dd90ae0f155d4d8ca9acf23b28109765 (table test-table [id=84f513ed5a4241eb8f3202c6c171109f]) successfully deleted
I20260619 14:52:41.557058 14365 ts_tablet_manager.cc:1905] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:41.558622 13821 catalog_manager.cc:4841] TS afaea075b0334bf3b1be3d55059e65bf (127.0.241.1:37175): tablet dd90ae0f155d4d8ca9acf23b28109765 (table test-table [id=84f513ed5a4241eb8f3202c6c171109f]) successfully deleted
I20260619 14:52:41.563884 14365 ts_tablet_manager.cc:1918] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:41.564113 14365 log.cc:1196] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/dd90ae0f155d4d8ca9acf23b28109765
I20260619 14:52:41.564697 14365 ts_tablet_manager.cc:1939] T dd90ae0f155d4d8ca9acf23b28109765 P 90b0c5fc5e44473da63377a601017ed8: Deleting consensus metadata
I20260619 14:52:41.564718 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.546590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.0.241.1:37175)
0619 14:52:41.546761 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:41.546767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:41.547079 (+ 312us) server_negotiation.cc:407] Connection header received
0619 14:52:41.547252 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:41.547258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:41.547353 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:41.547546 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:41.550072 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.551321 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.553213 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.553961 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.554110 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.557728 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:41.557739 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:41.557752 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:41.557781 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:41.561513 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:41.562337 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:41.562346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:41.562350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:41.562493 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:41.562896 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:41.562901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:41.562961 (+ 60us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:41.563473 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:41.563761 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:41.564104 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 14:52:41.564355 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:52:41.565560 14027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:41.566476 13820 catalog_manager.cc:4841] TS 90b0c5fc5e44473da63377a601017ed8 (127.0.241.3:38775): tablet dd90ae0f155d4d8ca9acf23b28109765 (table test-table [id=84f513ed5a4241eb8f3202c6c171109f]) successfully deleted
I20260619 14:52:41.667146 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 13872
I20260619 14:52:41.685789 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 14036
I20260619 14:52:41.698506 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.686523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.0.241.2:42765)
0619 14:52:41.686679 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:41.686683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:41.686701 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:52:41.686819 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:41.686823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:41.686891 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:41.686975 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:41.688344 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.689135 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.690470 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.690837 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.690923 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.693399 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:41.693411 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:41.693426 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:41.693455 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:41.696129 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:41.696786 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:41.696793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:41.696795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:41.696924 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:41.697266 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:41.697269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:41.697271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:41.697637 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:41.697862 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:41.698078 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 14:52:41.698225 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:52:41.699179 14172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:41.793565 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 14036
I20260619 14:52:41.810729 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 14181
I20260619 14:52:41.823710 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:41.811579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.0.241.3:38775)
0619 14:52:41.811745 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:41.811749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:41.811768 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:52:41.811898 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:41.811902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:41.811965 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:41.812051 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:41.813410 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.814197 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.815407 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:41.815894 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:41.815990 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:41.818616 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:41.818626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:41.818640 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:41.818676 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:41.821352 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:41.821967 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:41.821974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:41.821976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:41.822092 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:41.822378 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:41.822381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:41.822383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:41.822710 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:41.822920 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:41.823162 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 14:52:41.823324 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:52:41.824407 14306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:41.912356 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 14181
I20260619 14:52:41.928853 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 13802
I20260619 14:52:41.929970 13865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:42.041406 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 13802
I20260619 14:52:42.064122 964 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:52:42.400352 964 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:52:42.400550 964 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:52:42.746281 964 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:52:42.746 UTC [12854] LOG: received smart shutdown request
2026-06-19 14:52:42.749 UTC [12854] LOG: background worker "logical replication launcher" (PID 12863) exited with exit code 1
2026-06-19 14:52:42.750 UTC [12858] LOG: shutting down
2026-06-19 14:52:42.914 UTC [12854] LOG: database system is shut down
2026-06-19T14:52:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52667 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:52:43 dist-test-slave-p3ml krb5kdc[14378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:52:43 dist-test-slave-p3ml krb5kdc[14378](info): set up 2 sockets
Jun 19 14:52:43 dist-test-slave-p3ml krb5kdc[14378](info): commencing operation
krb5kdc: starting...
W20260619 14:52:44.152177 964 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:52:44 dist-test-slave-p3ml krb5kdc[14378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:52:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:52:44.379273 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33303
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:32915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:33303,127.0.241.61:44807,127.0.241.60:37583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:44.705976 14391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:44.706264 14391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:44.706364 14391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:44.713620 14391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:44.713732 14391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:44.713778 14391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:44.713841 14391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:44.713886 14391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:44.723875 14391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:32915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:33303,127.0.241.61:44807,127.0.241.60:37583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:33303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:44.725108 14391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:44.726356 14391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:44.738090 14399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:44.738194 14396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:44.738176 14397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:44.739749 14391 server_base.cc:1034] running on GCE node
I20260619 14:52:44.740638 14391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:44.742151 14391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:44.743513 14391 hybrid_clock.cc:648] HybridClock initialized: now 1781880764743394 us; error 41 us; skew 500 ppm
I20260619 14:52:44.743902 14391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:44 dist-test-slave-p3ml krb5kdc[14378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:44.753687 14391 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:52:44.755230 14391 webserver.cc:460] Webserver started at http://127.0.241.62:36877/ using document root <none> and password file <none>
I20260619 14:52:44.755962 14391 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:44.756068 14391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:44.756320 14391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:44.759232 14391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "4357876faddb4abf959f01ca260ae016"
format_stamp: "Formatted at 2026-06-19 14:52:44 on dist-test-slave-p3ml"
server_key: "78613054310ff6bad6f63cd16e85ab29"
server_key_iv: "59501496ce0ef41911fbe4f1485539df"
server_key_version: "encryptionkey@0"
I20260619 14:52:44.760077 14391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "4357876faddb4abf959f01ca260ae016"
format_stamp: "Formatted at 2026-06-19 14:52:44 on dist-test-slave-p3ml"
server_key: "78613054310ff6bad6f63cd16e85ab29"
server_key_iv: "59501496ce0ef41911fbe4f1485539df"
server_key_version: "encryptionkey@0"
I20260619 14:52:44.765074 14391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 14:52:44.768368 14406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:44.769657 14391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:52:44.769793 14391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "4357876faddb4abf959f01ca260ae016"
format_stamp: "Formatted at 2026-06-19 14:52:44 on dist-test-slave-p3ml"
server_key: "78613054310ff6bad6f63cd16e85ab29"
server_key_iv: "59501496ce0ef41911fbe4f1485539df"
server_key_version: "encryptionkey@0"
I20260619 14:52:44.769950 14391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:44.789630 14391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:44.794031 14391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:44.794267 14391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:44.813628 14391 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:33303
I20260619 14:52:44.813664 14457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:33303 every 8 connection(s)
I20260619 14:52:44.814992 14391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:44.818663 14458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:44.822109 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 14391
I20260619 14:52:44.822329 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:52:44.823140 964 external_mini_cluster.cc:1428] Setting key 524b1a7e1b25dc90fcdc16fb44af8103
I20260619 14:52:44.827165 14458 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:52:44 dist-test-slave-p3ml krb5kdc[14378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.241.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.61@KRBTEST.COM" created.
I20260619 14:52:44.850057 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:44.830879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33303 (local address 127.0.0.1:44914)
0619 14:52:44.831657 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:44.831699 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:44.833766 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:44.834560 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:44.834567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:44.835245 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:44.835649 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:44.835667 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:44.837248 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:44.837254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:44.838310 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:44.838322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:44.839250 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:44.839257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:44.839470 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:44.840193 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:44.840214 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:44.842614 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:44.845747 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:44.845758 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:44.845769 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:44.846217 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:44.846619 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:44.846626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:44.846630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:44.846785 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:44.847654 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:44.847666 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:44.847889 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 14:52:44.848133 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 14:52:44.848713 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":155,"thread_start_us":332,"threads_started":1}
I20260619 14:52:44.850036 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:44.832237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.0.241.62:33303)
0619 14:52:44.832989 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:44.833003 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:44.833050 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:52:44.834122 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:44.834128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:44.834340 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:44.834543 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:44.835840 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:44.837090 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:44.838456 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:44.839004 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:44.839212 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:44.842823 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:44.842830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:44.842841 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:44.842871 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:44.845609 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:44.846355 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:44.846371 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:44.846374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:44.846505 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:44.846920 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:44.846925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:44.846928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:44.847527 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:44.847731 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:44.848251 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 14:52:44.848616 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":524,"thread_start_us":353,"threads_started":1}
I20260619 14:52:44.853791 14458 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:52:44.854959 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:44.854457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39072)
0619 14:52:44.854655 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:44.854787 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 14:52:44.855265 14407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
I20260619 14:52:44.856905 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:44.856562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39084)
0619 14:52:44.856702 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:44.856775 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 14:52:44.857115 14458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:44807: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.61.keytab.
Entry for principal kudu/127.0.241.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.61.keytab.
I20260619 14:52:44.876174 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.61:44807
--webserver_interface=127.0.241.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.61.keytab
--principal=kudu/127.0.241.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:32915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:33303,127.0.241.61:44807,127.0.241.60:37583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:52:44.906445 14458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } attempt: 1
I20260619 14:52:44.907758 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:44.907186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39094)
0619 14:52:44.907448 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:44.907580 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 14:52:44.908869 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:44.908465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39102)
0619 14:52:44.908614 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:44.908695 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 14:52:44.909065 14458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:44807: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
I20260619 14:52:44.977365 14458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } attempt: 2
I20260619 14:52:44.978631 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:44.978051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39118)
0619 14:52:44.978329 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:44.978451 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 14:52:44.979837 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:44.979383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39128)
0619 14:52:44.979577 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:44.979679 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260619 14:52:44.980079 14458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:44807: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
I20260619 14:52:45.071363 14458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } attempt: 3
I20260619 14:52:45.072644 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.072200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39144)
0619 14:52:45.072411 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.072514 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 14:52:45.073884 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.073393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39160)
0619 14:52:45.073581 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.073697 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 14:52:45.074052 14458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:44807: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
W20260619 14:52:45.199199 14465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:45.199563 14465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:45.199728 14465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:45.207012 14465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:45.207160 14465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:45.207279 14465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:45.207391 14465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:45.207494 14465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:45.217298 14458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } attempt: 4
I20260619 14:52:45.217350 14465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:32915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:33303,127.0.241.61:44807,127.0.241.60:37583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.61.keytab
--principal=kudu/127.0.241.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.61:44807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.241.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:45.218420 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.217910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39168)
0619 14:52:45.218209 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.218294 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260619 14:52:45.218771 14465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:45.219532 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.219101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39180)
0619 14:52:45.219249 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.219345 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:52:45.219720 14458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:44807: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44807: connect: Connection refused (error 111)
I20260619 14:52:45.220192 14465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:45.231972 14470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:45.232014 14473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:45.232056 14471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:45.232831 14465 server_base.cc:1034] running on GCE node
I20260619 14:52:45.233575 14465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:45.234903 14465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:45.236162 14465 hybrid_clock.cc:648] HybridClock initialized: now 1781880765236118 us; error 60 us; skew 500 ppm
I20260619 14:52:45.236575 14465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:45 dist-test-slave-p3ml krb5kdc[14378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:45.239871 14465 init.cc:377] Logged in from keytab as kudu/127.0.241.61@KRBTEST.COM (short username kudu)
I20260619 14:52:45.241384 14465 webserver.cc:460] Webserver started at http://127.0.241.61:35977/ using document root <none> and password file <none>
I20260619 14:52:45.242100 14465 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:45.242260 14465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:45.242588 14465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:45.245705 14465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/data/instance:
uuid: "5ebcae60be95428f9fe6bba0222e1a32"
format_stamp: "Formatted at 2026-06-19 14:52:45 on dist-test-slave-p3ml"
server_key: "90170059219f5364f54cd424c3b804e7"
server_key_iv: "b7a319cc487db9266230bd12f50a2d25"
server_key_version: "encryptionkey@0"
I20260619 14:52:45.246555 14465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/wal/instance:
uuid: "5ebcae60be95428f9fe6bba0222e1a32"
format_stamp: "Formatted at 2026-06-19 14:52:45 on dist-test-slave-p3ml"
server_key: "90170059219f5364f54cd424c3b804e7"
server_key_iv: "b7a319cc487db9266230bd12f50a2d25"
server_key_version: "encryptionkey@0"
I20260619 14:52:45.251886 14465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 14:52:45.255249 14480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:45.256726 14465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:52:45.256918 14465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/wal
uuid: "5ebcae60be95428f9fe6bba0222e1a32"
format_stamp: "Formatted at 2026-06-19 14:52:45 on dist-test-slave-p3ml"
server_key: "90170059219f5364f54cd424c3b804e7"
server_key_iv: "b7a319cc487db9266230bd12f50a2d25"
server_key_version: "encryptionkey@0"
I20260619 14:52:45.257164 14465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:45.280346 14465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:45.285039 14465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:45.285347 14465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:45.304687 14465 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.61:44807
I20260619 14:52:45.304721 14531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.61:44807 every 8 connection(s)
I20260619 14:52:45.306221 14465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
I20260619 14:52:45.307965 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 14465
I20260619 14:52:45.308169 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-1/wal/instance
I20260619 14:52:45.308612 964 external_mini_cluster.cc:1428] Setting key ba3d2a730bb5794edf66fe0ee9922ecd
I20260619 14:52:45.310000 14532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:45.316434 14532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.241.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.60@KRBTEST.COM" created.
Jun 19 14:52:45 dist-test-slave-p3ml krb5kdc[14378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.61@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:45.337754 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.318952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44920 (local address 127.0.241.62:33303)
0619 14:52:45.319167 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.319174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.320485 (+ 1311us) server_negotiation.cc:407] Connection header received
0619 14:52:45.322350 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.322359 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.322453 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.322571 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.324060 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.325285 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.326774 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.327516 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.327665 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.331724 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.331729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.331732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.331767 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.334562 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.335353 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.335358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.335362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.335546 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.335977 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.335982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.335986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.336284 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.336444 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.337176 (+ 732us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.337415 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:52:45.338745 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.319502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33303 (local address 127.0.0.1:44920)
0619 14:52:45.320311 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.320356 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:45.322084 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:45.322765 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:45.322772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:45.323270 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:45.323882 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.323898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.325441 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.325445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.326618 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.326630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.327720 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.327725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.327907 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.328649 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:45.328674 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:45.331484 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:45.334729 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.334738 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.334757 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.335206 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.335662 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.335666 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.335670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.335855 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.336416 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:45.336424 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:45.336804 (+ 380us) client_negotiation.cc:769] Sending connection context
0619 14:52:45.337077 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 14:52:45.337610 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":367,"threads_started":1}
I20260619 14:52:45.340639 14532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:52:45 dist-test-slave-p3ml krb5kdc[14378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.61@KRBTEST.COM for kudu/127.0.241.61@KRBTEST.COM
I20260619 14:52:45.355885 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.341453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39196)
0619 14:52:45.341661 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.341681 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:45.341794 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:45.342849 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:45.342854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:45.343284 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:45.343421 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.343481 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.344807 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.344815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.345640 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.345650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.346463 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.346468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.346669 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.347262 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:45.347282 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:45.349490 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:45.353007 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.353014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.353021 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.353455 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.353906 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.353911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.353914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.354043 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.354735 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:45.354742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:45.354942 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 14:52:45.355113 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 14:52:45.355454 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 14:52:45.355953 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.341643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.0.241.61:44807)
0619 14:52:45.342457 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.342462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.342488 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:52:45.342529 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.342534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.342700 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.342857 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.343620 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.344663 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.345786 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.346320 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.346456 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.349688 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.349699 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.349711 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.349741 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.352864 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.353594 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.353605 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.353609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.353785 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.354158 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.354162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.354166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.354596 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.354861 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.355246 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.355582 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":677,"thread_start_us":482,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.60.keytab.
Entry for principal kudu/127.0.241.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.60.keytab.
I20260619 14:52:45.358527 14532 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:52:45.359130 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.60:37583
--webserver_interface=127.0.241.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.60.keytab
--principal=kudu/127.0.241.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:32915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:33303,127.0.241.61:44807,127.0.241.60:37583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:52:45.359797 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.359247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50718)
0619 14:52:45.359510 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.359638 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 14:52:45.360133 14483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
I20260619 14:52:45.361728 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.361313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50720)
0619 14:52:45.361447 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.361573 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:52:45.362056 14532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37583: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
I20260619 14:52:45.411387 14532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } attempt: 1
I20260619 14:52:45.412835 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.412266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50724)
0619 14:52:45.412511 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.412645 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 14:52:45.413941 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.413544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50738)
0619 14:52:45.413711 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.413811 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 14:52:45.414181 14532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37583: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
I20260619 14:52:45.482456 14532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } attempt: 2
I20260619 14:52:45.483865 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.483135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50740)
0619 14:52:45.483425 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.483635 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 14:52:45.484987 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.484595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50742)
0619 14:52:45.484756 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.484845 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:52:45.485232 14532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37583: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
I20260619 14:52:45.519006 14458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } attempt: 5
Jun 19 14:52:45 dist-test-slave-p3ml krb5kdc[14378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for kudu/127.0.241.61@KRBTEST.COM
I20260619 14:52:45.533068 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.519774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39208)
0619 14:52:45.519969 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.519991 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:45.520141 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:45.520568 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:45.520575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:45.520981 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:45.521157 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.521166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.522179 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.522184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.523242 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.523254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.524181 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.524187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.524389 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.525051 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:45.525068 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:45.527505 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:45.530553 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.530570 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.530575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.530935 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.531336 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.531341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.531345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.531547 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.532076 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:45.532080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:45.532239 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 14:52:45.532394 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 14:52:45.532700 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 14:52:45.533070 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.520008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.0.241.61:44807)
0619 14:52:45.520205 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.520211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.520233 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:52:45.520349 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.520354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.520413 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.520528 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.521319 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.522038 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.523396 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.524032 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.524170 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.527690 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.527695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.527698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.527729 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.530386 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.531082 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.531088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.531091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.531215 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.531654 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.531660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.531663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.531946 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.532096 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.532513 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.532726 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:52:45.534268 14458 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:52:45.535429 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.535049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50754)
0619 14:52:45.535215 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.535293 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 14:52:45.536408 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.536090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50762)
0619 14:52:45.536231 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.536319 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:52:45.536594 14458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37583: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
I20260619 14:52:45.576483 14532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } attempt: 3
I20260619 14:52:45.577636 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.577185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50772)
0619 14:52:45.577394 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.577480 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:52:45.578677 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.578262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50786)
0619 14:52:45.578488 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.578562 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260619 14:52:45.578972 14532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37583: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
I20260619 14:52:45.587839 14458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } attempt: 1
I20260619 14:52:45.588964 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.588518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50792)
0619 14:52:45.588699 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.588812 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 14:52:45.589953 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.589647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50808)
0619 14:52:45.589802 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.589862 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:52:45.590142 14458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37583: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
I20260619 14:52:45.658392 14458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } attempt: 2
I20260619 14:52:45.659641 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.659079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50810)
0619 14:52:45.659333 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.659419 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 14:52:45.660845 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.660453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50818)
0619 14:52:45.660652 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.660724 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260619 14:52:45.661092 14458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37583: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
W20260619 14:52:45.686561 14539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:45.686918 14539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:45.687076 14539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:45.695188 14539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:45.695361 14539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:45.695506 14539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:45.695616 14539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:45.695708 14539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:45.706085 14539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:32915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:33303,127.0.241.61:44807,127.0.241.60:37583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.60.keytab
--principal=kudu/127.0.241.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.60:37583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.241.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:45.707610 14539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:45.709038 14539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:45.721864 14544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:45.722151 14532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } attempt: 4
W20260619 14:52:45.721908 14547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:45.721967 14545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:45.723582 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.722953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50820)
0619 14:52:45.723206 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.723332 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 14:52:45.723548 14539 server_base.cc:1034] running on GCE node
I20260619 14:52:45.724256 14539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:45.724547 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.724223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50834)
0619 14:52:45.724375 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.724452 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 14:52:45.724800 14532 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37583: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
I20260619 14:52:45.725682 14539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:45.726907 14539 hybrid_clock.cc:648] HybridClock initialized: now 1781880765726859 us; error 33 us; skew 500 ppm
I20260619 14:52:45.727238 14539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:45 dist-test-slave-p3ml krb5kdc[14378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:45.730270 14539 init.cc:377] Logged in from keytab as kudu/127.0.241.60@KRBTEST.COM (short username kudu)
I20260619 14:52:45.731820 14539 webserver.cc:460] Webserver started at http://127.0.241.60:37537/ using document root <none> and password file <none>
I20260619 14:52:45.732443 14539 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:45.732537 14539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:45.732776 14539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:45.735707 14539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/data/instance:
uuid: "79d321d96d76400f941ef0c5ea273d3f"
format_stamp: "Formatted at 2026-06-19 14:52:45 on dist-test-slave-p3ml"
server_key: "9954f70905d70cb186677f18f7cbe7b1"
server_key_iv: "e09c649b35c13213efc39d045a87e917"
server_key_version: "encryptionkey@0"
I20260619 14:52:45.736445 14539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/wal/instance:
uuid: "79d321d96d76400f941ef0c5ea273d3f"
format_stamp: "Formatted at 2026-06-19 14:52:45 on dist-test-slave-p3ml"
server_key: "9954f70905d70cb186677f18f7cbe7b1"
server_key_iv: "e09c649b35c13213efc39d045a87e917"
server_key_version: "encryptionkey@0"
I20260619 14:52:45.741400 14539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:52:45.744817 14554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:45.746163 14539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:52:45.746325 14539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/wal
uuid: "79d321d96d76400f941ef0c5ea273d3f"
format_stamp: "Formatted at 2026-06-19 14:52:45 on dist-test-slave-p3ml"
server_key: "9954f70905d70cb186677f18f7cbe7b1"
server_key_iv: "e09c649b35c13213efc39d045a87e917"
server_key_version: "encryptionkey@0"
I20260619 14:52:45.746527 14539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:45.766697 14539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:45.767328 14458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } attempt: 3
I20260619 14:52:45.768348 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.767925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50842)
0619 14:52:45.768100 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.768190 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 14:52:45.769335 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.768978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50844)
0619 14:52:45.769174 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.769245 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260619 14:52:45.769570 14458 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37583: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37583: connect: Connection refused (error 111)
I20260619 14:52:45.771123 14539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:45.771414 14539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:45.789812 14539 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.60:37583
I20260619 14:52:45.789947 14605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.60:37583 every 8 connection(s)
I20260619 14:52:45.791190 14539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
I20260619 14:52:45.791756 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 14539
I20260619 14:52:45.791960 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880565277815-964-0/minicluster-data/master-2/wal/instance
I20260619 14:52:45.792577 964 external_mini_cluster.cc:1428] Setting key b37edd232ffd269bac4d5532dde1cd9b
I20260619 14:52:45.795552 14606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:45.802317 14606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:52:45 dist-test-slave-p3ml krb5kdc[14378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.60@KRBTEST.COM
Jun 19 14:52:45 dist-test-slave-p3ml krb5kdc[14378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.60@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:45.818137 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.794504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.0.241.60:37583)
0619 14:52:45.795276 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.795288 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.795334 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:52:45.797175 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.797201 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.797567 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.797996 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.799729 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.801134 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.802726 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.803550 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.803769 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.808381 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.808391 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.808405 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.808459 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.813130 (+ 4671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.814041 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.814051 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.814059 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.814236 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.814731 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.814737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.814754 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.815408 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.815691 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.816611 (+ 920us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.817003 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":343,"threads_started":1}
I20260619 14:52:45.822225 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.804915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.0.241.62:33303)
0619 14:52:45.805121 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.805128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.805599 (+ 471us) server_negotiation.cc:407] Connection header received
0619 14:52:45.805773 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.805782 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.805877 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.805991 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.807003 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.807811 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.809970 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.810758 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.810870 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.815324 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.815331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.815336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.815374 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.818993 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.819846 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.819857 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.819859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.820004 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.820553 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.820559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.820566 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.820922 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.821088 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.821657 (+ 569us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.821868 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 14:52:45.822324 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.804699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:33303 (local address 127.0.0.1:44932)
0619 14:52:45.805454 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.805491 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:45.805633 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:45.806050 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:45.806054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:45.806555 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:45.806793 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.806801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.807981 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.807987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.809753 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.809772 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.811190 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.811197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.811412 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.812160 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:45.812191 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:45.815105 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:45.819144 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.819153 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.819157 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.819698 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.820139 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.820151 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.820154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.820416 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.821082 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:45.821093 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:45.821310 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 14:52:45.821551 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 14:52:45.821844 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":414,"threads_started":1}
I20260619 14:52:45.824185 14606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:52:45 dist-test-slave-p3ml krb5kdc[14378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.60@KRBTEST.COM for kudu/127.0.241.61@KRBTEST.COM
Jun 19 14:52:45 dist-test-slave-p3ml krb5kdc[14378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:45.838651 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.824958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44807 (local address 127.0.0.1:39216)
0619 14:52:45.825205 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.825219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:45.825305 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:45.825624 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:45.825627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:45.825965 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:45.826101 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.826108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.827113 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.827119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.828392 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.828404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.829218 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.829224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.829427 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.829936 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:45.829952 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:45.832224 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:45.835873 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.835877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.835880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.836419 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.836972 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.836988 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.836992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.837225 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.837722 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:45.837727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:45.837926 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 14:52:45.838095 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 14:52:45.838391 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260619 14:52:45.838742 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.824883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.0.241.61:44807)
0619 14:52:45.825046 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.825050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.825297 (+ 247us) server_negotiation.cc:407] Connection header received
0619 14:52:45.825397 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.825405 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.825482 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.825586 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.826246 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.826966 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.828537 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.829063 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.829192 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.832409 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.832416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.832420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.832456 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.835748 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.836564 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.836571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.836609 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.836819 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.837352 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.837356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.837358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.837599 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.837709 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.838176 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.838369 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:52:45.839793 14606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:52:45.840119 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.822410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.0.241.62:33303)
0619 14:52:45.823150 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.823158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.823185 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:52:45.823359 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.823364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.823454 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.823564 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.825179 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.826222 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.828119 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.828560 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.828686 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.833665 (+ 4979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.833672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.833678 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.833715 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.837056 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.837832 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.837836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.837838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.837952 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.838373 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.838377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.838380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.838728 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.838995 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.839535 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.839743 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":372,"threads_started":1}
Jun 19 14:52:45 dist-test-slave-p3ml krb5kdc[14378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.60@KRBTEST.COM for kudu/127.0.241.60@KRBTEST.COM
Jun 19 14:52:45 dist-test-slave-p3ml krb5kdc[14378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.61@KRBTEST.COM
I20260619 14:52:45.853665 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.840538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50866)
0619 14:52:45.840830 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.840849 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:45.840973 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:45.841286 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:45.841290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:45.841727 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:45.841845 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.841853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.843262 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.843266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.844342 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.844355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.845182 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.845186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.845332 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.845901 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:45.845917 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:45.848121 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:45.851023 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.851030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.851036 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.851610 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.851969 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.851973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.851977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.852147 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.852741 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:45.852745 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:45.852938 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 14:52:45.853118 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 14:52:45.853369 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":2}
I20260619 14:52:45.854019 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.840576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.0.241.60:37583)
0619 14:52:45.840732 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.840735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.840921 (+ 186us) server_negotiation.cc:407] Connection header received
0619 14:52:45.841096 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.841101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.841167 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.841273 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.841994 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.843128 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.844491 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.845018 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.845172 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.848282 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.848288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.848293 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.848328 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.850767 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.851716 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.851720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.851722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.851832 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.852273 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.852278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.852284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.852616 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.852784 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.853235 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.853575 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":52}
I20260619 14:52:45.858037 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.841822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.0.241.61:44807)
0619 14:52:45.842007 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.842012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.842089 (+ 77us) server_negotiation.cc:407] Connection header received
0619 14:52:45.842220 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.842223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.842272 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.842342 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.843990 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.844835 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.846508 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.847141 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.847233 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.852185 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.852189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.852192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.852249 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.855599 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.856343 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.856347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.856349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.856472 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.856864 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.856867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.856869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.857125 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.857319 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.857636 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.857786 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:52:45.860008 14606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f: Bootstrap starting.
I20260619 14:52:45.863088 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:52:45.863940 14606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:45.865162 14606 log.cc:824] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:45.868413 14606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f: No bootstrap required, opened a new log
I20260619 14:52:45.879848 14606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } }
I20260619 14:52:45.880451 14606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:45.880630 14606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d321d96d76400f941ef0c5ea273d3f, State: Initialized, Role: FOLLOWER
I20260619 14:52:45.881698 14606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } }
I20260619 14:52:45.890431 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.868595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.0.241.62:33303)
0619 14:52:45.868829 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.868837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.869389 (+ 552us) server_negotiation.cc:407] Connection header received
0619 14:52:45.869603 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.869608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.869685 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.869784 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.873665 (+ 3881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.875011 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.877790 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.878477 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.878600 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.883098 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.883104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.883108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.883152 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.886840 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.887864 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.887869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.887873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.888035 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.888542 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.888549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.888555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.888959 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.889218 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.889799 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.890045 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:52:45.891835 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.869105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.0.241.61:44807)
0619 14:52:45.869270 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.869275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.870260 (+ 985us) server_negotiation.cc:407] Connection header received
0619 14:52:45.870407 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.870413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.870490 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.870583 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.872849 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.873968 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.877856 (+ 3888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.878492 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.878589 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.883099 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.883109 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.883116 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.883183 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.887620 (+ 4437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.888666 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.888674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.888680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.888880 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.889430 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.889438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.889444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.889926 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.890215 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.890942 (+ 727us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.891249 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:52:45.893601 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.870785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.0.241.60:37583)
0619 14:52:45.871009 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.871017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.876413 (+ 5396us) server_negotiation.cc:407] Connection header received
0619 14:52:45.876596 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.876599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.876656 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.876759 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.878509 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.879595 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.882103 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.882882 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.882984 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.886726 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.886732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.886737 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.886779 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.890319 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.891319 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.891324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.891327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.891530 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.892024 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.892028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.892031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.892371 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.892595 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.893142 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.893361 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 14:52:45.894990 14620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } } }
I20260619 14:52:45.895499 14620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:45.896027 14606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:52:45.905323 14631 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:52:45.905586 14631 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:52:45.902515 14539 master_runner.cc:186] Error getting master registration for 127.0.241.62:33303: OK, instance_id { permanent_uuid: "4357876faddb4abf959f01ca260ae016" instance_seqno: 1781880764808105 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:52:45.911548 14539 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:52:45.946821 14458 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } attempt: 4
Jun 19 14:52:45 dist-test-slave-p3ml krb5kdc[14378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for kudu/127.0.241.60@KRBTEST.COM
I20260619 14:52:45.963063 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.947551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.0.241.60:37583)
0619 14:52:45.947726 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:45.947731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:45.947753 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:52:45.947933 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:45.947937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:45.948021 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:45.948126 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:45.949152 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.950123 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.952016 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.952535 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.952639 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.957429 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:45.957434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:45.957437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:45.957467 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:45.960443 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.961231 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.961238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.961246 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.961397 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:45.961800 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:45.961803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:45.961805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:45.962096 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:45.962199 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:45.962617 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 14:52:45.962797 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:52:45.963228 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:45.947442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50870)
0619 14:52:45.947683 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:45.947703 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:45.947808 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:45.948185 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:45.948190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:45.948705 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:45.948912 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.948924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.950275 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.950281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.951885 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:45.951900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:45.952689 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:45.952695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:45.952872 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:45.954466 (+ 1594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:45.954490 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:45.957250 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:45.960563 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.960568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.960572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.961089 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.961521 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:45.961527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:45.961531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:45.961687 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:45.962214 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:45.962219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:45.962376 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 14:52:45.962530 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 14:52:45.962847 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 14:52:45.969589 14458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016: Bootstrap starting.
I20260619 14:52:45.973512 14458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:45.974754 14458 log.cc:824] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:45.977875 14458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016: No bootstrap required, opened a new log
I20260619 14:52:45.989576 14458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } }
I20260619 14:52:45.989982 14458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:45.990099 14458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4357876faddb4abf959f01ca260ae016, State: Initialized, Role: FOLLOWER
I20260619 14:52:45.990761 14458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } }
I20260619 14:52:45.991755 14634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } } }
I20260619 14:52:45.992122 14634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:45.993120 14458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:52:46.004209 14645 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:52:46.004426 14645 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:52:46.003104 14391 master_runner.cc:186] Error getting master registration for 127.0.241.61:44807: OK, instance_id { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" instance_seqno: 1781880765299393 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:52:46.016353 14391 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 14:52:46.023476 14560 tablet.cc:2369] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:46.023998 14532 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } attempt: 5
Jun 19 14:52:46 dist-test-slave-p3ml krb5kdc[14378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.61@KRBTEST.COM for kudu/127.0.241.60@KRBTEST.COM
I20260619 14:52:46.042537 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:46.024600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37583 (local address 127.0.0.1:50884)
0619 14:52:46.024861 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:46.024882 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:46.025012 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:46.025407 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:46.025412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:46.025940 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:46.026149 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:46.026161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:46.027651 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:46.027657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:46.029092 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:46.029107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:46.031334 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:46.031342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:46.031618 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:46.032319 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:46.032343 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:46.035230 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:46.039667 (+ 4437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:46.039674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:46.039678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:46.040238 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:46.040655 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:46.040660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:46.040663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:46.040827 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:46.041466 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:46.041472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:46.041628 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 14:52:46.041786 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 14:52:46.042074 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 14:52:46.042596 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:46.024689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.0.241.60:37583)
0619 14:52:46.024891 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:46.024897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:46.024946 (+ 49us) server_negotiation.cc:407] Connection header received
0619 14:52:46.025149 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:46.025155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:46.025234 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:46.025355 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:46.026310 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:46.027519 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:46.030376 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:46.031087 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:46.031215 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:46.035467 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:46.035474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:46.035479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:46.035518 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:46.039518 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:46.040378 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:46.040384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:46.040387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:46.040532 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:46.040949 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:46.040955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:46.040958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:46.041332 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:46.041530 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:46.041935 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 14:52:46.042213 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 14:52:46.047041 14412 tablet.cc:2369] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:46.050531 14532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32: Bootstrap starting.
I20260619 14:52:46.054664 14532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:46.056016 14532 log.cc:824] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:46.059238 14532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32: No bootstrap required, opened a new log
I20260619 14:52:46.070264 14532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } }
I20260619 14:52:46.070662 14532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:46.070788 14532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ebcae60be95428f9fe6bba0222e1a32, State: Initialized, Role: FOLLOWER
I20260619 14:52:46.071449 14532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } }
I20260619 14:52:46.072408 14648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } } }
I20260619 14:52:46.072919 14648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:46.073691 14532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:46.092691 14465 master_runner.cc:263] No leader master found from master 5ebcae60be95428f9fe6bba0222e1a32
I20260619 14:52:46.092855 14465 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 14:52:46.098023 14659 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:52:46.098155 14659 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:52:46.186228 14620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:46.186610 14620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } }
I20260619 14:52:46.188580 14620 leader_election.cc:288] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4357876faddb4abf959f01ca260ae016 (127.0.241.62:33303), 5ebcae60be95428f9fe6bba0222e1a32 (127.0.241.61:44807)
I20260619 14:52:46.189963 14433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79d321d96d76400f941ef0c5ea273d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4357876faddb4abf959f01ca260ae016" is_pre_election: true
I20260619 14:52:46.190626 14433 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79d321d96d76400f941ef0c5ea273d3f in term 0.
I20260619 14:52:46.191502 14558 leader_election.cc:302] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4357876faddb4abf959f01ca260ae016, 79d321d96d76400f941ef0c5ea273d3f; no voters:
I20260619 14:52:46.192636 14507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79d321d96d76400f941ef0c5ea273d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ebcae60be95428f9fe6bba0222e1a32" is_pre_election: true
I20260619 14:52:46.193271 14507 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79d321d96d76400f941ef0c5ea273d3f in term 0.
I20260619 14:52:46.198164 14620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:46.198333 14620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:46.198457 14620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:46.202371 14620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } }
I20260619 14:52:46.203649 14433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79d321d96d76400f941ef0c5ea273d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4357876faddb4abf959f01ca260ae016"
I20260619 14:52:46.203888 14433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:46.207329 14433 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79d321d96d76400f941ef0c5ea273d3f in term 1.
I20260619 14:52:46.207931 14558 leader_election.cc:302] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4357876faddb4abf959f01ca260ae016, 79d321d96d76400f941ef0c5ea273d3f; no voters:
I20260619 14:52:46.211455 14507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79d321d96d76400f941ef0c5ea273d3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ebcae60be95428f9fe6bba0222e1a32"
I20260619 14:52:46.211720 14507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:46.215250 14507 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79d321d96d76400f941ef0c5ea273d3f in term 1.
I20260619 14:52:46.216109 14620 leader_election.cc:288] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [CANDIDATE]: Term 1 election: Requested vote from peers 4357876faddb4abf959f01ca260ae016 (127.0.241.62:33303), 5ebcae60be95428f9fe6bba0222e1a32 (127.0.241.61:44807)
I20260619 14:52:46.216567 14620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:46.217006 14620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [term 1 LEADER]: Becoming Leader. State: Replica: 79d321d96d76400f941ef0c5ea273d3f, State: Running, Role: LEADER
I20260619 14:52:46.217855 14620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } }
I20260619 14:52:46.242095 14661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79d321d96d76400f941ef0c5ea273d3f. Latest consensus state: current_term: 1 leader_uuid: "79d321d96d76400f941ef0c5ea273d3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } } }
I20260619 14:52:46.242702 14661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:46.251950 14663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:46.254236 14663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:46.280613 14507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [term 1 FOLLOWER]: Refusing update from remote peer 79d321d96d76400f941ef0c5ea273d3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:46.280656 14433 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [term 1 FOLLOWER]: Refusing update from remote peer 79d321d96d76400f941ef0c5ea273d3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:46.281579 14661 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:46.282282 14620 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:52:46.287966 14486 tablet.cc:2369] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:46.325452 14648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79d321d96d76400f941ef0c5ea273d3f. Latest consensus state: current_term: 1 leader_uuid: "79d321d96d76400f941ef0c5ea273d3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } } }
I20260619 14:52:46.325752 14648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:46.328759 14664 mvcc.cc:205] Tried to move back new op lower bound from 7298583618675023872 to 7298583618441687040. Current Snapshot: MvccSnapshot[applied={T|T < 7298583618675023872}]
I20260619 14:52:46.332345 14666 mvcc.cc:205] Tried to move back new op lower bound from 7298583618675023872 to 7298583618441687040. Current Snapshot: MvccSnapshot[applied={T|T < 7298583618675023872}]
I20260619 14:52:46.335983 14648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79d321d96d76400f941ef0c5ea273d3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } } }
I20260619 14:52:46.336230 14648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:46.344412 14634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79d321d96d76400f941ef0c5ea273d3f. Latest consensus state: current_term: 1 leader_uuid: "79d321d96d76400f941ef0c5ea273d3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } } }
I20260619 14:52:46.344707 14634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:46.349092 14661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79d321d96d76400f941ef0c5ea273d3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } } }
I20260619 14:52:46.349380 14661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:46.354408 14620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79d321d96d76400f941ef0c5ea273d3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } } }
I20260619 14:52:46.355132 14620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:46.357714 14665 mvcc.cc:205] Tried to move back new op lower bound from 7298583618675023872 to 7298583618441687040. Current Snapshot: MvccSnapshot[applied={T|T < 7298583618675023872}]
I20260619 14:52:46.360883 14663 catalog_manager.cc:1347] Generated new cluster ID: 509a501bad8c45608db81beda412b674
I20260619 14:52:46.360988 14663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:46.368417 14634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79d321d96d76400f941ef0c5ea273d3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4357876faddb4abf959f01ca260ae016" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 33303 } } peers { permanent_uuid: "5ebcae60be95428f9fe6bba0222e1a32" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44807 } } peers { permanent_uuid: "79d321d96d76400f941ef0c5ea273d3f" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37583 } } }
I20260619 14:52:46.368698 14634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:46.410270 14663 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:46.412161 14663 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:46.428937 14663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79d321d96d76400f941ef0c5ea273d3f: Generated new TSK 0
I20260619 14:52:46.430099 14663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:52:47.006364 14645 catalog_manager.cc:1259] Loaded cluster ID: 509a501bad8c45608db81beda412b674
I20260619 14:52:47.006465 14645 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016: loading cluster ID for follower catalog manager: success
I20260619 14:52:47.008863 14645 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016: acquiring CA information for follower catalog manager: success
I20260619 14:52:47.010567 14645 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4357876faddb4abf959f01ca260ae016: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:52:47.100274 14659 catalog_manager.cc:1259] Loaded cluster ID: 509a501bad8c45608db81beda412b674
I20260619 14:52:47.100416 14659 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32: loading cluster ID for follower catalog manager: success
I20260619 14:52:47.102900 14659 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32: acquiring CA information for follower catalog manager: success
I20260619 14:52:47.104792 14659 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ebcae60be95428f9fe6bba0222e1a32: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:52:47.515578 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 14391
I20260619 14:52:47.516937 14453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:47.628741 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 14391
I20260619 14:52:47.649648 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 14465
I20260619 14:52:47.650763 14527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:47.756676 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 14465
I20260619 14:52:47.776863 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 14539
I20260619 14:52:47.778003 14601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:47.887946 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 14539
2026-06-19T14:52:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4848 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:52:47 dist-test-slave-p3ml krb5kdc[14682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:52:47 dist-test-slave-p3ml krb5kdc[14682](info): set up 2 sockets
Jun 19 14:52:47 dist-test-slave-p3ml krb5kdc[14682](info): commencing operation
krb5kdc: starting...
W20260619 14:52:48.995999 964 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:52:49 dist-test-slave-p3ml krb5kdc[14682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:52:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:52:49.231312 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:44547
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:44547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:49.562428 14695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:49.562801 14695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:49.563005 14695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:49.573941 14695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:49.574092 14695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:49.574196 14695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:49.574304 14695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:49.574396 14695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:49.585244 14695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:44547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:49.586619 14695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:49.588006 14695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:49.600488 14700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:49.600556 14701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:49.600560 14703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:49.601308 14695 server_base.cc:1034] running on GCE node
I20260619 14:52:49.602061 14695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:49.603506 14695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:49.604730 14695 hybrid_clock.cc:648] HybridClock initialized: now 1781880769604672 us; error 54 us; skew 500 ppm
I20260619 14:52:49.605037 14695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:49 dist-test-slave-p3ml krb5kdc[14682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:49.608243 14695 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:52:49.609761 14695 webserver.cc:460] Webserver started at http://127.0.241.62:34257/ using document root <none> and password file <none>
I20260619 14:52:49.610445 14695 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:49.610579 14695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:49.610916 14695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:49.613940 14695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "3d4c056dd58648d4bfe22605cc552d5d"
format_stamp: "Formatted at 2026-06-19 14:52:49 on dist-test-slave-p3ml"
server_key: "331875dc8813618cd625ff3759b9ea47"
server_key_iv: "e9b670ded4c960198eb708dd98b18930"
server_key_version: "encryptionkey@0"
I20260619 14:52:49.614698 14695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "3d4c056dd58648d4bfe22605cc552d5d"
format_stamp: "Formatted at 2026-06-19 14:52:49 on dist-test-slave-p3ml"
server_key: "331875dc8813618cd625ff3759b9ea47"
server_key_iv: "e9b670ded4c960198eb708dd98b18930"
server_key_version: "encryptionkey@0"
I20260619 14:52:49.619796 14695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 14:52:49.623128 14710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:49.624500 14695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:52:49.624697 14695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "3d4c056dd58648d4bfe22605cc552d5d"
format_stamp: "Formatted at 2026-06-19 14:52:49 on dist-test-slave-p3ml"
server_key: "331875dc8813618cd625ff3759b9ea47"
server_key_iv: "e9b670ded4c960198eb708dd98b18930"
server_key_version: "encryptionkey@0"
I20260619 14:52:49.624958 14695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:49.647223 14695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:49.651265 14695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:49.651669 14695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:49.669530 14695 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:44547
I20260619 14:52:49.669572 14761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:44547 every 8 connection(s)
I20260619 14:52:49.670897 14695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:49.674191 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 14695
I20260619 14:52:49.674428 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:52:49.674805 14762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:49.675274 964 external_mini_cluster.cc:1428] Setting key 19325ff6a2394ba6fc0fd51d7393c06d
I20260619 14:52:49.687302 14762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d: Bootstrap starting.
Jun 19 14:52:49 dist-test-slave-p3ml krb5kdc[14682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:49.690855 14762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:49.692322 14762 log.cc:824] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:49.695389 14762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d: No bootstrap required, opened a new log
I20260619 14:52:49.697943 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:49.677134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.0.241.62:44547)
0619 14:52:49.677901 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:49.677912 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:49.677952 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:52:49.679703 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:49.679723 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:49.680103 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:49.680507 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:49.682038 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:49.683285 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:49.684641 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:49.685345 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:49.685564 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:49.690475 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:49.690483 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:49.690497 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:49.690544 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:49.693845 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:49.694566 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:49.694572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:49.694578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:49.694739 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:49.695141 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:49.695147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:49.695150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:49.695796 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:49.696141 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:49.696560 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 14:52:49.696969 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":333,"threads_started":1}
I20260619 14:52:49.701849 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:52:49.704712 14762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4c056dd58648d4bfe22605cc552d5d" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 44547 } }
I20260619 14:52:49.705202 14762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:49.705312 14762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d4c056dd58648d4bfe22605cc552d5d, State: Initialized, Role: FOLLOWER
I20260619 14:52:49.705996 14762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4c056dd58648d4bfe22605cc552d5d" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 44547 } }
I20260619 14:52:49.706259 14762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:49.706393 14762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:49.706568 14762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:49.708801 14762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4c056dd58648d4bfe22605cc552d5d" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 44547 } }
I20260619 14:52:49.709326 14762 leader_election.cc:302] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d4c056dd58648d4bfe22605cc552d5d; no voters:
I20260619 14:52:49.709913 14762 leader_election.cc:288] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:49.710166 14767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:49.710976 14767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [term 1 LEADER]: Becoming Leader. State: Replica: 3d4c056dd58648d4bfe22605cc552d5d, State: Running, Role: LEADER
I20260619 14:52:49.711539 14767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4c056dd58648d4bfe22605cc552d5d" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 44547 } }
I20260619 14:52:49.712219 14762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:49.714601 14769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d4c056dd58648d4bfe22605cc552d5d. Latest consensus state: current_term: 1 leader_uuid: "3d4c056dd58648d4bfe22605cc552d5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4c056dd58648d4bfe22605cc552d5d" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 44547 } } }
I20260619 14:52:49.714965 14769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:49.715730 14768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d4c056dd58648d4bfe22605cc552d5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4c056dd58648d4bfe22605cc552d5d" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 44547 } } }
I20260619 14:52:49.719947 14768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:49.720520 14774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:49.724831 14774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:49.734078 14774 catalog_manager.cc:1347] Generated new cluster ID: 1e1b3f02461e44609d95f59fa948035c
I20260619 14:52:49.734211 14774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:49.747751 14774 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:49.749146 14774 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:49.759711 14774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d4c056dd58648d4bfe22605cc552d5d: Generated new TSK 0
I20260619 14:52:49.760587 14774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:52:49.814908 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 14695
I20260619 14:52:49.816108 14757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:49.912863 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 14695
2026-06-19T14:52:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2020 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:52:50 dist-test-slave-p3ml krb5kdc[14790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:52:50 dist-test-slave-p3ml krb5kdc[14790](info): set up 2 sockets
Jun 19 14:52:50 dist-test-slave-p3ml krb5kdc[14790](info): commencing operation
krb5kdc: starting...
W20260619 14:52:51.019764 964 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:52:51 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:52:51.252832 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45075
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:51.582644 14803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:51.582906 14803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:51.583084 14803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:51.590441 14803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:51.590548 14803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:51.590679 14803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:51.590775 14803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:51.590842 14803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:51.600744 14803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:45075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:51.602063 14803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:51.603335 14803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:51.614836 14811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:51.614990 14808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:51.614916 14809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:51.616014 14803 server_base.cc:1034] running on GCE node
I20260619 14:52:51.616750 14803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:51.618121 14803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:51.619354 14803 hybrid_clock.cc:648] HybridClock initialized: now 1781880771619322 us; error 61 us; skew 500 ppm
I20260619 14:52:51.619726 14803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:51 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:51.622776 14803 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:52:51.624253 14803 webserver.cc:460] Webserver started at http://127.0.241.62:39741/ using document root <none> and password file <none>
I20260619 14:52:51.624912 14803 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:51.625062 14803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:51.625365 14803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:51.628327 14803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "15aefc63f510414c9c7361697cb7b778"
format_stamp: "Formatted at 2026-06-19 14:52:51 on dist-test-slave-p3ml"
server_key: "bf9345d004627addda117a26d6840eb8"
server_key_iv: "bff440e7dfb65d39434677ee00af2f30"
server_key_version: "encryptionkey@0"
I20260619 14:52:51.629093 14803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "15aefc63f510414c9c7361697cb7b778"
format_stamp: "Formatted at 2026-06-19 14:52:51 on dist-test-slave-p3ml"
server_key: "bf9345d004627addda117a26d6840eb8"
server_key_iv: "bff440e7dfb65d39434677ee00af2f30"
server_key_version: "encryptionkey@0"
I20260619 14:52:51.633944 14803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:52:51.637439 14818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:51.638696 14803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:52:51.638886 14803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "15aefc63f510414c9c7361697cb7b778"
format_stamp: "Formatted at 2026-06-19 14:52:51 on dist-test-slave-p3ml"
server_key: "bf9345d004627addda117a26d6840eb8"
server_key_iv: "bff440e7dfb65d39434677ee00af2f30"
server_key_version: "encryptionkey@0"
I20260619 14:52:51.639087 14803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:51.650714 14803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:51.654970 14803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:51.655265 14803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:51.673808 14803 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:45075
I20260619 14:52:51.673893 14869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:45075 every 8 connection(s)
I20260619 14:52:51.675168 14803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:51.678886 14870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:51.685158 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 14803
I20260619 14:52:51.685374 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:52:51.686031 964 external_mini_cluster.cc:1428] Setting key 95b96ffa2e4850f7f03b500cfcae2492
I20260619 14:52:51.690523 14870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Bootstrap starting.
I20260619 14:52:51.694342 14870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:51.695493 14870 log.cc:824] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:51.698663 14870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: No bootstrap required, opened a new log
Jun 19 14:52:51 dist-test-slave-p3ml krb5kdc[14790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:51.710002 14870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } }
I20260619 14:52:51.710482 14870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:51.710613 14870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15aefc63f510414c9c7361697cb7b778, State: Initialized, Role: FOLLOWER
I20260619 14:52:51.710655 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:51.688090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.0.241.62:45075)
0619 14:52:51.688905 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:51.688916 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:51.688954 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:52:51.690553 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:51.690573 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:51.690989 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:51.691376 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:51.692819 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:51.694661 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:51.696163 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:51.697076 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:51.697282 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:51.701545 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:51.701556 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:51.701574 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:51.701632 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:51.705970 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:51.706732 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:51.706742 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:51.706750 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:51.706964 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:51.707332 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:51.707343 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:51.707346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:51.708175 (+ 829us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:51.708524 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:51.708831 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 14:52:51.709285 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":396,"threads_started":1}
I20260619 14:52:51.711272 14870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } }
I20260619 14:52:51.711676 14870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:51.711851 14870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:51.712054 14870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:51.714604 14870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } }
I20260619 14:52:51.715104 14870 leader_election.cc:302] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15aefc63f510414c9c7361697cb7b778; no voters:
I20260619 14:52:51.715873 14870 leader_election.cc:288] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:51.715963 14875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:51.716820 14875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 1 LEADER]: Becoming Leader. State: Replica: 15aefc63f510414c9c7361697cb7b778, State: Running, Role: LEADER
I20260619 14:52:51.717687 14875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } }
I20260619 14:52:51.718259 14870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:51.720577 14876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15aefc63f510414c9c7361697cb7b778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } } }
I20260619 14:52:51.720552 14877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15aefc63f510414c9c7361697cb7b778. Latest consensus state: current_term: 1 leader_uuid: "15aefc63f510414c9c7361697cb7b778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } } }
I20260619 14:52:51.722450 14877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:51.723095 14876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:51.723306 14884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:51.728190 14884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:51.737454 14884 catalog_manager.cc:1347] Generated new cluster ID: a1adaba70d7c4562b16f2e4c630ae838
I20260619 14:52:51.737573 14884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:51.772967 14884 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:51.774853 14884 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:51.781101 14884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Generated new TSK 0
I20260619 14:52:51.781898 14884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:52:51.852977 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45075
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:52.214533 14896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:52.214785 14896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:52.214877 14896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:52.222862 14896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:52.222970 14896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:52.223083 14896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:52:52.232892 14896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:52.234086 14896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:52.235289 14896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:52.248560 14901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:52.248483 14902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:52.248454 14904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:52.249392 14896 server_base.cc:1034] running on GCE node
I20260619 14:52:52.250102 14896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:52.250993 14896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:52.252197 14896 hybrid_clock.cc:648] HybridClock initialized: now 1781880772252139 us; error 58 us; skew 500 ppm
I20260619 14:52:52.252549 14896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:52 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:52.255843 14896 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:52:52.257198 14896 webserver.cc:460] Webserver started at http://127.0.241.1:46303/ using document root <none> and password file <none>
I20260619 14:52:52.257828 14896 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:52.257928 14896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:52.258170 14896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:52.261096 14896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "4f5646b30e734d74aa51a90b5205eeec"
format_stamp: "Formatted at 2026-06-19 14:52:52 on dist-test-slave-p3ml"
server_key: "9d6635aeb47604ccba69111b7c87981b"
server_key_iv: "ac5647daa96685c47e284ef9dfb19a10"
server_key_version: "encryptionkey@0"
I20260619 14:52:52.261808 14896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "4f5646b30e734d74aa51a90b5205eeec"
format_stamp: "Formatted at 2026-06-19 14:52:52 on dist-test-slave-p3ml"
server_key: "9d6635aeb47604ccba69111b7c87981b"
server_key_iv: "ac5647daa96685c47e284ef9dfb19a10"
server_key_version: "encryptionkey@0"
I20260619 14:52:52.266917 14896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:52:52.270532 14911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:52.271935 14896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:52:52.272126 14896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "4f5646b30e734d74aa51a90b5205eeec"
format_stamp: "Formatted at 2026-06-19 14:52:52 on dist-test-slave-p3ml"
server_key: "9d6635aeb47604ccba69111b7c87981b"
server_key_iv: "ac5647daa96685c47e284ef9dfb19a10"
server_key_version: "encryptionkey@0"
I20260619 14:52:52.272341 14896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:52.292135 14896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:52.297164 14896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:52.297500 14896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:52.298493 14896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:52.299981 14896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:52.300081 14896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:52.300230 14896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:52.300300 14896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:52.333340 14896 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:45447
I20260619 14:52:52.333384 15023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:45447 every 8 connection(s)
I20260619 14:52:52.334640 14896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:52.340297 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 14896
I20260619 14:52:52.340534 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:52:52.341310 964 external_mini_cluster.cc:1428] Setting key b74c1f849e5c2ee690433b3156adb231
Jun 19 14:52:52 dist-test-slave-p3ml krb5kdc[14790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:52.355837 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:52.337195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:59369 (local address 127.0.241.62:45075)
0619 14:52:52.337649 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:52.337653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:52.338676 (+ 1023us) server_negotiation.cc:407] Connection header received
0619 14:52:52.340665 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:52.340671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:52.340768 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:52.340909 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:52.342646 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:52.343773 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:52.345344 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:52.346084 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:52.346254 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:52.349664 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:52.349669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:52.349673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:52.349704 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:52.352814 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:52.353546 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:52.353554 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:52.353558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:52.353678 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:52.354104 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:52.354110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:52.354114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:52.354385 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:52.354543 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:52.355222 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 14:52:52.355470 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
I20260619 14:52:52.356760 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:52.337780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:59369)
0619 14:52:52.338507 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:52.338548 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:52.340341 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:52.341103 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:52.341111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:52.341696 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:52.342439 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:52.342464 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:52.343927 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:52.343931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:52.345170 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:52.345181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:52.346304 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:52.346309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:52.346504 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:52.347182 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:52.347208 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:52.349431 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:52.352953 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:52.352962 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:52.352979 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:52.353412 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:52.353789 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:52.353794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:52.353798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:52.353994 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:52.354520 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:52.354528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:52.354858 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 14:52:52.355121 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 14:52:52.355678 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":277,"threads_started":1}
I20260619 14:52:52.358492 15024 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45075
I20260619 14:52:52.358801 15024 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:52.359571 15024 heartbeater.cc:511] Master 127.0.241.62:45075 requested a full tablet report, sending...
I20260619 14:52:52.362897 14835 ts_manager.cc:194] Registered new tserver with Master: 4f5646b30e734d74aa51a90b5205eeec (127.0.241.1:45447)
I20260619 14:52:52.364459 14835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:59369
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:52:52.398613 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45075
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:52.730979 15030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:52.731230 15030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:52.731323 15030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:52.738567 15030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:52.738687 15030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:52.738801 15030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:52:52.748749 15030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:52.750002 15030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:52.751296 15030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:52.764472 15038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:52.764535 15036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:52.764678 15035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:52.765486 15030 server_base.cc:1034] running on GCE node
I20260619 14:52:52.766148 15030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:52.767035 15030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:52.768255 15030 hybrid_clock.cc:648] HybridClock initialized: now 1781880772768222 us; error 54 us; skew 500 ppm
I20260619 14:52:52.768565 15030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:52 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:52.771749 15030 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:52:52.773033 15030 webserver.cc:460] Webserver started at http://127.0.241.2:41263/ using document root <none> and password file <none>
I20260619 14:52:52.773646 15030 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:52.773744 15030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:52.773978 15030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:52.776921 15030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1"
format_stamp: "Formatted at 2026-06-19 14:52:52 on dist-test-slave-p3ml"
server_key: "55a996edc14d7ce0f0f685b0eefb9d5a"
server_key_iv: "4d39f1e225c6444b53811e2e2cf62c7f"
server_key_version: "encryptionkey@0"
I20260619 14:52:52.777666 15030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1"
format_stamp: "Formatted at 2026-06-19 14:52:52 on dist-test-slave-p3ml"
server_key: "55a996edc14d7ce0f0f685b0eefb9d5a"
server_key_iv: "4d39f1e225c6444b53811e2e2cf62c7f"
server_key_version: "encryptionkey@0"
I20260619 14:52:52.782759 15030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 14:52:52.786087 15045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:52.787315 15030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:52:52.787511 15030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1"
format_stamp: "Formatted at 2026-06-19 14:52:52 on dist-test-slave-p3ml"
server_key: "55a996edc14d7ce0f0f685b0eefb9d5a"
server_key_iv: "4d39f1e225c6444b53811e2e2cf62c7f"
server_key_version: "encryptionkey@0"
I20260619 14:52:52.787724 15030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:52.824697 15030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:52.828907 15030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:52.829193 15030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:52.830168 15030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:52.831589 15030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:52.831660 15030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:52.831840 15030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:52.831892 15030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:52.866039 15030 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:40675
I20260619 14:52:52.866122 15157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:40675 every 8 connection(s)
I20260619 14:52:52.867291 15030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:52.872146 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 15030
I20260619 14:52:52.872328 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:52:52.872782 964 external_mini_cluster.cc:1428] Setting key 7f83bcc7eb6756cadadcaf9ac4d1b770
Jun 19 14:52:52 dist-test-slave-p3ml krb5kdc[14790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:52.888748 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:52.869787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:44797 (local address 127.0.241.62:45075)
0619 14:52:52.870242 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:52.870248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:52.871235 (+ 987us) server_negotiation.cc:407] Connection header received
0619 14:52:52.873185 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:52.873191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:52.873278 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:52.873392 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:52.875322 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:52.876636 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:52.878254 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:52.879037 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:52.879184 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:52.882931 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:52.882936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:52.882939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:52.882971 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:52.885617 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:52.886349 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:52.886357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:52.886362 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:52.886508 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:52.886930 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:52.886935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:52.886938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:52.887209 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:52.887371 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:52.888116 (+ 745us) server_negotiation.cc:299] Negotiation successful
0619 14:52:52.888352 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":211,"threads_started":1}
I20260619 14:52:52.889721 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:52.870381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:44797)
0619 14:52:52.871053 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:52.871117 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:52.872906 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:52.873780 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:52.873789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:52.874326 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:52.875133 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:52.875153 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:52.876771 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:52.876775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:52.878111 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:52.878123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:52.879241 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:52.879246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:52.879487 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:52.880169 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:52.880195 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:52.882705 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:52.885765 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:52.885775 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:52.885792 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:52.886224 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:52.886622 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:52.886628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:52.886632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:52.886815 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:52.887342 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:52.887350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:52.887757 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 14:52:52.888035 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 14:52:52.888563 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":268,"threads_started":1}
I20260619 14:52:52.891300 15158 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45075
I20260619 14:52:52.891672 15158 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:52.892598 15158 heartbeater.cc:511] Master 127.0.241.62:45075 requested a full tablet report, sending...
I20260619 14:52:52.894337 14835 ts_manager.cc:194] Registered new tserver with Master: dccd7fd581cb43aa968a4e8f8ad0b4f1 (127.0.241.2:40675)
I20260619 14:52:52.895632 14835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:44797
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:52:52.926263 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45075
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:53.243649 15164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:53.243988 15164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:53.244150 15164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:53.251345 15164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:53.251536 15164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:53.251714 15164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:52:53.261497 15164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:53.262849 15164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:53.264196 15164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:53.277684 15169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:53.277627 15170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:53.277740 15172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:53.278523 15164 server_base.cc:1034] running on GCE node
I20260619 14:52:53.279121 15164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:53.280098 15164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:53.281349 15164 hybrid_clock.cc:648] HybridClock initialized: now 1781880773281298 us; error 41 us; skew 500 ppm
I20260619 14:52:53.281713 15164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:53 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:53.285092 15164 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:52:53.286705 15164 webserver.cc:460] Webserver started at http://127.0.241.3:43225/ using document root <none> and password file <none>
I20260619 14:52:53.287402 15164 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:53.287626 15164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:53.287995 15164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:53.290995 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "41da6d08489742bbbc2a5cc12946e6a7"
format_stamp: "Formatted at 2026-06-19 14:52:53 on dist-test-slave-p3ml"
server_key: "260a942338b6ebc56f7b00c1cc004515"
server_key_iv: "e6e3cd7f38ccc8581a41cc03c8c2cb14"
server_key_version: "encryptionkey@0"
I20260619 14:52:53.291875 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "41da6d08489742bbbc2a5cc12946e6a7"
format_stamp: "Formatted at 2026-06-19 14:52:53 on dist-test-slave-p3ml"
server_key: "260a942338b6ebc56f7b00c1cc004515"
server_key_iv: "e6e3cd7f38ccc8581a41cc03c8c2cb14"
server_key_version: "encryptionkey@0"
I20260619 14:52:53.297163 15164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:52:53.300705 15179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:53.302054 15164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 14:52:53.302232 15164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "41da6d08489742bbbc2a5cc12946e6a7"
format_stamp: "Formatted at 2026-06-19 14:52:53 on dist-test-slave-p3ml"
server_key: "260a942338b6ebc56f7b00c1cc004515"
server_key_iv: "e6e3cd7f38ccc8581a41cc03c8c2cb14"
server_key_version: "encryptionkey@0"
I20260619 14:52:53.302433 15164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:53.329769 15164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:53.334234 15164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:53.334542 15164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:53.335685 15164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:53.337459 15164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:53.337546 15164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:53.337647 15164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:53.337703 15164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:53.367395 15024 heartbeater.cc:503] Master 127.0.241.62:45075 was elected leader, sending a full tablet report...
I20260619 14:52:53.369882 15164 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:38665
I20260619 14:52:53.369964 15291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:38665 every 8 connection(s)
I20260619 14:52:53.371100 15164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:53.377681 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 15164
I20260619 14:52:53.377867 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:52:53.378345 964 external_mini_cluster.cc:1428] Setting key 0c20be09129cc1ef45512aebe62a6f3f
Jun 19 14:52:53 dist-test-slave-p3ml krb5kdc[14790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:53.390394 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.373649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:47857 (local address 127.0.241.62:45075)
0619 14:52:53.373850 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:53.373854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:53.375157 (+ 1303us) server_negotiation.cc:407] Connection header received
0619 14:52:53.377046 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:53.377050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:53.377116 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:53.377221 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:53.378771 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.379678 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.381411 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.381802 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.381899 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.385258 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:53.385263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:53.385266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:53.385298 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:53.387801 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:53.388471 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:53.388474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:53.388476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:53.388599 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:53.388980 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:53.388982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:53.388985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:53.389240 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:53.389346 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:53.389998 (+ 652us) server_negotiation.cc:299] Negotiation successful
0619 14:52:53.390165 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:52:53.391969 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.374174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:47857)
0619 14:52:53.375004 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:53.375048 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:53.376798 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:53.377403 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:53.377411 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:53.377995 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:53.378601 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:53.378620 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.379848 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.379852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:53.381291 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:53.381303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.382026 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.382032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:53.382225 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.382828 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:53.382850 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:53.385063 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:53.387924 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:53.387931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:53.387944 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:53.388368 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:53.388696 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:53.388699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:53.388701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:53.388889 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:53.389347 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:53.389352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:53.389672 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 14:52:53.389909 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 14:52:53.390292 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":355,"threads_started":1}
I20260619 14:52:53.393503 15292 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45075
I20260619 14:52:53.393812 15292 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:53.394497 15292 heartbeater.cc:511] Master 127.0.241.62:45075 requested a full tablet report, sending...
I20260619 14:52:53.395937 14835 ts_manager.cc:194] Registered new tserver with Master: 41da6d08489742bbbc2a5cc12946e6a7 (127.0.241.3:38665)
I20260619 14:52:53.396708 14835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:47857
I20260619 14:52:53.405797 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:52:53.422689 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.410247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.0.241.62:45075)
0619 14:52:53.410487 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:53.410493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:53.410847 (+ 354us) server_negotiation.cc:407] Connection header received
0619 14:52:53.410982 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:53.410986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:53.411060 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:53.411140 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:53.412642 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.413394 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.414644 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.415036 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.415122 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.417571 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:53.417575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:53.417578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:53.417606 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:53.420361 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:53.421062 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:53.421066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:53.421068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:53.421180 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:53.421535 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:53.421538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:53.421540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:53.421820 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:53.422001 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:53.422287 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 14:52:53.422443 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:52:53.426684 14835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:52:53.433485 14835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:52:53.459575 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.446319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45447 (local address 127.0.0.1:49396)
0619 14:52:53.448612 (+ 2293us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:53.448635 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:53.448751 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:53.449837 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:53.449848 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:53.449889 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:53.450417 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:53.450434 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.452668 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.452677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:53.455789 (+ 3112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:53.455804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.458318 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.458327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:53.458628 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.458771 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 14:52:53.458929 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 14:52:53.459119 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2129,"spinlock_wait_cycles":8064,"thread_start_us":2108,"threads_started":1}
I20260619 14:52:53.459969 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.446820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49396 (local address 127.0.241.1:45447)
0619 14:52:53.448162 (+ 1342us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:53.448174 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:53.448871 (+ 697us) server_negotiation.cc:407] Connection header received
0619 14:52:53.448970 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:53.448978 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:53.449233 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:53.449459 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:53.450674 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.452480 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.455972 (+ 3492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.457710 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.457867 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.458161 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:53.459272 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0619 14:52:53.459455 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":1406,"threads_started":1}
I20260619 14:52:53.463928 14959 tablet_service.cc:1467] Processing CreateTablet for tablet 49a32dcffdae4e44a7d17ca0e86b9860 (DEFAULT_TABLE table=test-table [id=35a82c144e9f4377bc609ab4751505b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:53.465453 14959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a32dcffdae4e44a7d17ca0e86b9860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:53.466351 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.445696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:38665 (local address 127.0.0.1:55466)
0619 14:52:53.447058 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:53.447111 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:53.447308 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:53.459892 (+ 12584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:53.459897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:53.459923 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:53.460280 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:53.460292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.462484 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.462489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:53.464105 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:53.464117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.465495 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.465504 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:53.465721 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.465753 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:52:53.465852 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:52:53.465983 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"mutex_wait_us":1,"thread_start_us":882,"threads_started":1}
I20260619 14:52:53.466398 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.445868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.0.241.3:38665)
0619 14:52:53.459315 (+ 13447us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:53.459324 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:53.459352 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:52:53.459478 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:53.459485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:53.459663 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:53.460498 (+ 835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:53.460763 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.462343 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.464254 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.465286 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.465412 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.465632 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:53.465914 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 14:52:53.466072 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13278,"thread_start_us":13114,"threads_started":1}
I20260619 14:52:53.468418 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.447471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:40675 (local address 127.0.0.1:56372)
0619 14:52:53.459012 (+ 11541us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:53.459038 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:53.459164 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:53.460952 (+ 1788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:53.460962 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:53.460986 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:53.461442 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:53.461520 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.462930 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.462935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:53.465029 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:53.465045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.467655 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.467661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:53.467830 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.467870 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 14:52:53.467973 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:52:53.468094 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11390,"thread_start_us":11222,"threads_started":1}
I20260619 14:52:53.468483 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.448023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.0.241.2:40675)
0619 14:52:53.458322 (+ 10299us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:53.458334 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:53.459300 (+ 966us) server_negotiation.cc:407] Connection header received
0619 14:52:53.459409 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:53.459418 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:53.459669 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:53.460721 (+ 1052us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:53.461671 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.462781 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.465205 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.467451 (+ 2246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.467613 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.467803 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:53.468013 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 14:52:53.468181 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10071,"thread_start_us":10295,"threads_started":1}
I20260619 14:52:53.470359 15227 tablet_service.cc:1467] Processing CreateTablet for tablet 49a32dcffdae4e44a7d17ca0e86b9860 (DEFAULT_TABLE table=test-table [id=35a82c144e9f4377bc609ab4751505b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:53.471243 15093 tablet_service.cc:1467] Processing CreateTablet for tablet 49a32dcffdae4e44a7d17ca0e86b9860 (DEFAULT_TABLE table=test-table [id=35a82c144e9f4377bc609ab4751505b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:53.471863 15227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a32dcffdae4e44a7d17ca0e86b9860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:53.472405 15093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49a32dcffdae4e44a7d17ca0e86b9860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:53.484277 15310 tablet_bootstrap.cc:492] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec: Bootstrap starting.
I20260619 14:52:53.486434 15311 tablet_bootstrap.cc:492] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Bootstrap starting.
I20260619 14:52:53.488153 15310 tablet_bootstrap.cc:654] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:53.488998 15312 tablet_bootstrap.cc:492] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7: Bootstrap starting.
I20260619 14:52:53.489521 15310 log.cc:824] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:53.490188 15311 tablet_bootstrap.cc:654] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:53.491608 15311 log.cc:824] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:53.492379 15312 tablet_bootstrap.cc:654] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:53.492947 15310 tablet_bootstrap.cc:492] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec: No bootstrap required, opened a new log
I20260619 14:52:53.493230 15310 ts_tablet_manager.cc:1397] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 14:52:53.493568 15312 log.cc:824] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:53.494771 15311 tablet_bootstrap.cc:492] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1: No bootstrap required, opened a new log
I20260619 14:52:53.495029 15311 ts_tablet_manager.cc:1397] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 14:52:53.501964 15312 tablet_bootstrap.cc:492] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7: No bootstrap required, opened a new log
I20260619 14:52:53.502223 15312 ts_tablet_manager.cc:1397] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260619 14:52:53.502151 15311 raft_consensus.cc:348] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:53.502557 15311 raft_consensus.cc:374] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:53.502661 15311 raft_consensus.cc:729] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccd7fd581cb43aa968a4e8f8ad0b4f1, State: Initialized, Role: FOLLOWER
I20260619 14:52:53.503240 15311 consensus_queue.cc:260] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:53.504328 15158 heartbeater.cc:503] Master 127.0.241.62:45075 was elected leader, sending a full tablet report...
I20260619 14:52:53.505028 15311 ts_tablet_manager.cc:1428] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260619 14:52:53.504886 15310 raft_consensus.cc:348] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:53.505280 15310 raft_consensus.cc:374] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:53.505358 15310 raft_consensus.cc:729] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5646b30e734d74aa51a90b5205eeec, State: Initialized, Role: FOLLOWER
I20260619 14:52:53.505957 15310 consensus_queue.cc:260] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:53.508656 15310 ts_tablet_manager.cc:1428] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260619 14:52:53.512848 15312 raft_consensus.cc:348] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:53.513172 15312 raft_consensus.cc:374] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:53.513245 15312 raft_consensus.cc:729] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41da6d08489742bbbc2a5cc12946e6a7, State: Initialized, Role: FOLLOWER
I20260619 14:52:53.513664 15312 consensus_queue.cc:260] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:53.514544 15292 heartbeater.cc:503] Master 127.0.241.62:45075 was elected leader, sending a full tablet report...
I20260619 14:52:53.515156 15312 ts_tablet_manager.cc:1428] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
W20260619 14:52:53.587196 15025 tablet.cc:2369] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:52:53.619266 15159 tablet.cc:2369] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:52:53.622891 15293 tablet.cc:2369] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:53.657050 15317 raft_consensus.cc:482] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:53.657348 15317 raft_consensus.cc:504] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:53.659083 15317 leader_election.cc:288] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41da6d08489742bbbc2a5cc12946e6a7 (127.0.241.3:38665), dccd7fd581cb43aa968a4e8f8ad0b4f1 (127.0.241.2:40675)
I20260619 14:52:53.666260 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.659545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:50067 (local address 127.0.241.3:38665)
0619 14:52:53.659767 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:53.659776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:53.660556 (+ 780us) server_negotiation.cc:407] Connection header received
0619 14:52:53.660642 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:53.660647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:53.660727 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:53.660822 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:53.661595 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.662458 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.664391 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.665284 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.665379 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.665451 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:53.665874 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 14:52:53.666004 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:52:53.666487 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.660564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:50283 (local address 127.0.241.2:40675)
0619 14:52:53.660753 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:53.660761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:53.661083 (+ 322us) server_negotiation.cc:407] Connection header received
0619 14:52:53.661228 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:53.661233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:53.661313 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:53.661418 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:53.661987 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.662817 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.664655 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.665530 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.665664 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.665754 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:53.666097 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 14:52:53.666223 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:52:53.666604 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.659555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:38665 (local address 127.0.241.1:50067)
0619 14:52:53.660223 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:53.660239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:53.660421 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:53.660970 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:53.660973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:53.660994 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:53.661430 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:53.661443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.662598 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.662603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:53.664239 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:53.664250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.665496 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.665503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:53.665683 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.665719 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:52:53.665815 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:52:53.665906 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":1,"thread_start_us":363,"threads_started":1}
I20260619 14:52:53.666640 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.660379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:40675 (local address 127.0.241.1:50283)
0619 14:52:53.660969 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:53.660982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:53.661098 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:53.661440 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:53.661442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:53.661459 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:53.661827 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:53.661839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.662963 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.662966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:53.664491 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:53.664505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.665746 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.665750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:53.665927 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.665960 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:52:53.666067 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:52:53.666189 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":2,"thread_start_us":305,"threads_started":1}
I20260619 14:52:53.667676 15247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a32dcffdae4e44a7d17ca0e86b9860" candidate_uuid: "4f5646b30e734d74aa51a90b5205eeec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41da6d08489742bbbc2a5cc12946e6a7" is_pre_election: true
I20260619 14:52:53.667665 15113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a32dcffdae4e44a7d17ca0e86b9860" candidate_uuid: "4f5646b30e734d74aa51a90b5205eeec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" is_pre_election: true
I20260619 14:52:53.668165 15247 raft_consensus.cc:2454] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5646b30e734d74aa51a90b5205eeec in term 0.
I20260619 14:52:53.668202 15113 raft_consensus.cc:2454] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5646b30e734d74aa51a90b5205eeec in term 0.
I20260619 14:52:53.668804 14912 leader_election.cc:302] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5646b30e734d74aa51a90b5205eeec, dccd7fd581cb43aa968a4e8f8ad0b4f1; no voters:
I20260619 14:52:53.669196 15317 raft_consensus.cc:2781] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:53.669340 15317 raft_consensus.cc:482] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:53.669471 15317 raft_consensus.cc:3037] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:53.671784 15317 raft_consensus.cc:504] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:53.672350 15317 leader_election.cc:288] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [CANDIDATE]: Term 1 election: Requested vote from peers 41da6d08489742bbbc2a5cc12946e6a7 (127.0.241.3:38665), dccd7fd581cb43aa968a4e8f8ad0b4f1 (127.0.241.2:40675)
I20260619 14:52:53.672883 15247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a32dcffdae4e44a7d17ca0e86b9860" candidate_uuid: "4f5646b30e734d74aa51a90b5205eeec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41da6d08489742bbbc2a5cc12946e6a7"
I20260619 14:52:53.673002 15113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49a32dcffdae4e44a7d17ca0e86b9860" candidate_uuid: "4f5646b30e734d74aa51a90b5205eeec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1"
I20260619 14:52:53.673097 15247 raft_consensus.cc:3037] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:53.673290 15113 raft_consensus.cc:3037] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:53.675292 15247 raft_consensus.cc:2454] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5646b30e734d74aa51a90b5205eeec in term 1.
I20260619 14:52:53.675696 14912 leader_election.cc:302] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41da6d08489742bbbc2a5cc12946e6a7, 4f5646b30e734d74aa51a90b5205eeec; no voters:
I20260619 14:52:53.675997 15317 raft_consensus.cc:2781] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:53.676522 15317 raft_consensus.cc:686] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 1 LEADER]: Becoming Leader. State: Replica: 4f5646b30e734d74aa51a90b5205eeec, State: Running, Role: LEADER
I20260619 14:52:53.676488 15113 raft_consensus.cc:2454] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5646b30e734d74aa51a90b5205eeec in term 1.
I20260619 14:52:53.677380 15317 consensus_queue.cc:237] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:53.682065 14834 catalog_manager.cc:5495] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f5646b30e734d74aa51a90b5205eeec (127.0.241.1). New cstate: current_term: 1 leader_uuid: "4f5646b30e734d74aa51a90b5205eeec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } health_report { overall_health: HEALTHY } } }
I20260619 14:52:53.750213 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.745164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.0.241.1:45447)
0619 14:52:53.745342 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:53.745346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:53.745382 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:52:53.745588 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:53.745592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:53.745668 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:53.745789 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:53.746137 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.746836 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.748248 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.748646 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.748737 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.749163 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:53.749739 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:53.749846 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:53.749955 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 14:52:53.750019 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:52:53.758087 15247 raft_consensus.cc:1261] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7 [term 1 FOLLOWER]: Refusing update from remote peer 4f5646b30e734d74aa51a90b5205eeec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:53.758153 15113 raft_consensus.cc:1261] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 1 FOLLOWER]: Refusing update from remote peer 4f5646b30e734d74aa51a90b5205eeec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:53.759069 15321 consensus_queue.cc:1035] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [LEADER]: Connected to new peer: Peer: permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:53.759480 15317 consensus_queue.cc:1035] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [LEADER]: Connected to new peer: Peer: permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:53.770925 15324 mvcc.cc:205] Tried to move back new op lower bound from 7298583649303584768 to 7298583648985792512. Current Snapshot: MvccSnapshot[applied={T|T < 7298583649303584768}]
I20260619 14:52:53.772555 15326 mvcc.cc:205] Tried to move back new op lower bound from 7298583649303584768 to 7298583648985792512. Current Snapshot: MvccSnapshot[applied={T|T < 7298583649303584768}]
I20260619 14:52:53.784614 14835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:53.784919 14835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:53.788552 14835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Sending DeleteTablet for 3 replicas of tablet 49a32dcffdae4e44a7d17ca0e86b9860
I20260619 14:52:53.789793 15227 tablet_service.cc:1514] Processing DeleteTablet for tablet 49a32dcffdae4e44a7d17ca0e86b9860 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:53 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:55466
I20260619 14:52:53.789851 14959 tablet_service.cc:1514] Processing DeleteTablet for tablet 49a32dcffdae4e44a7d17ca0e86b9860 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:53 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:49396
I20260619 14:52:53.790422 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 14896
I20260619 14:52:53.791332 15093 tablet_service.cc:1514] Processing DeleteTablet for tablet 49a32dcffdae4e44a7d17ca0e86b9860 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:53 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:56372
I20260619 14:52:53.792712 15335 tablet_replica.cc:330] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec: stopping tablet replica
I20260619 14:52:53.793397 15335 raft_consensus.cc:2229] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:52:53.794148 15335 raft_consensus.cc:2258] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:53.796710 15335 ts_tablet_manager.cc:1905] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:53.806011 15335 ts_tablet_manager.cc:1918] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:53.806250 15335 log.cc:1196] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/49a32dcffdae4e44a7d17ca0e86b9860
I20260619 14:52:53.807070 15335 ts_tablet_manager.cc:1939] T 49a32dcffdae4e44a7d17ca0e86b9860 P 4f5646b30e734d74aa51a90b5205eeec: Deleting consensus metadata
I20260619 14:52:53.809873 14821 catalog_manager.cc:4841] TS 4f5646b30e734d74aa51a90b5205eeec (127.0.241.1:45447): tablet 49a32dcffdae4e44a7d17ca0e86b9860 (table test-table [id=35a82c144e9f4377bc609ab4751505b5]) successfully deleted
I20260619 14:52:53.811049 15336 tablet_replica.cc:330] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1: stopping tablet replica
I20260619 14:52:53.811595 15336 raft_consensus.cc:2229] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:53.812268 15336 raft_consensus.cc:2258] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:53.813969 15334 tablet_replica.cc:330] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7: stopping tablet replica
I20260619 14:52:53.814411 15334 raft_consensus.cc:2229] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:53.814958 15334 raft_consensus.cc:2258] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:53.815944 15336 ts_tablet_manager.cc:1905] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:53.817298 15334 ts_tablet_manager.cc:1905] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:53.824409 15336 ts_tablet_manager.cc:1918] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:53.824646 15336 log.cc:1196] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/49a32dcffdae4e44a7d17ca0e86b9860
I20260619 14:52:53.824903 15334 ts_tablet_manager.cc:1918] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:53.825089 15334 log.cc:1196] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/49a32dcffdae4e44a7d17ca0e86b9860
I20260619 14:52:53.825487 15336 ts_tablet_manager.cc:1939] T 49a32dcffdae4e44a7d17ca0e86b9860 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Deleting consensus metadata
I20260619 14:52:53.825660 15334 ts_tablet_manager.cc:1939] T 49a32dcffdae4e44a7d17ca0e86b9860 P 41da6d08489742bbbc2a5cc12946e6a7: Deleting consensus metadata
I20260619 14:52:53.827502 14819 catalog_manager.cc:4841] TS dccd7fd581cb43aa968a4e8f8ad0b4f1 (127.0.241.2:40675): tablet 49a32dcffdae4e44a7d17ca0e86b9860 (table test-table [id=35a82c144e9f4377bc609ab4751505b5]) successfully deleted
I20260619 14:52:53.827796 14819 catalog_manager.cc:4841] TS 41da6d08489742bbbc2a5cc12946e6a7 (127.0.241.3:38665): tablet 49a32dcffdae4e44a7d17ca0e86b9860 (table test-table [id=35a82c144e9f4377bc609ab4751505b5]) successfully deleted
Jun 19 14:52:53 dist-test-slave-p3ml krb5kdc[14790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:52:53.839752 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.793030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.0.241.1:45447)
0619 14:52:53.794296 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:53.794303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:53.813680 (+ 19377us) server_negotiation.cc:407] Connection header received
0619 14:52:53.813757 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:53.813763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:53.813867 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:53.814000 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:53.816355 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.817637 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.822519 (+ 4882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.823333 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.823495 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.831775 (+ 8280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:53.831796 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:53.831820 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:53.831890 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:53.836106 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:53.837029 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:53.837052 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:53.837056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:53.837258 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:53.837750 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:53.837755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:53.837758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:53.838204 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:53.838497 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:53.839087 (+ 590us) server_negotiation.cc:299] Negotiation successful
0619 14:52:53.839332 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:52:53.840982 15019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:53.960038 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 14896
I20260619 14:52:53.978156 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 15030
Jun 19 14:52:53 dist-test-slave-p3ml krb5kdc[14790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:52:53.993542 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:53.978941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.0.241.2:40675)
0619 14:52:53.979152 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:53.979158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:53.979219 (+ 61us) server_negotiation.cc:407] Connection header received
0619 14:52:53.979413 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:53.979419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:53.979536 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:53.979651 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:53.980858 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.981621 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.982964 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:53.983350 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:53.983536 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:53.987915 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:53.987937 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:53.987952 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:53.987991 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:53.990575 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:53.991276 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:53.991284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:53.991288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:53.991469 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:53.991849 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:53.991856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:53.991859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:53.992319 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:53.992620 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:53.992934 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 14:52:53.993183 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:52:53.994248 15153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:54.083758 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 15030
I20260619 14:52:54.100636 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 15164
Jun 19 14:52:54 dist-test-slave-p3ml krb5kdc[14790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:52:54.116395 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:54.101514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.0.241.3:38665)
0619 14:52:54.101696 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:54.101704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:54.101790 (+ 86us) server_negotiation.cc:407] Connection header received
0619 14:52:54.102004 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:54.102009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:54.102093 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:54.102204 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:54.103493 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:54.104228 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:54.105581 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:54.105965 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:54.106124 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:54.110831 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:54.110847 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:54.110863 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:54.110897 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:54.113508 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:54.114191 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:54.114198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:54.114200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:54.114337 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:54.114727 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:54.114733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:54.114736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:54.115101 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:54.115390 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:54.115808 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 14:52:54.116030 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:52:54.117084 15287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:54.206079 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 15164
I20260619 14:52:54.222864 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 14803
I20260619 14:52:54.224262 14865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:54.334636 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 14803
I20260619 14:52:54.355597 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45075
--webserver_interface=127.0.241.62
--webserver_port=39741
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:54.682165 15342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:54.682449 15342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:54.682538 15342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:54.690014 15342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:54.690130 15342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:54.690176 15342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:54.690234 15342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:54.690279 15342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:54.700358 15342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:45075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:54.701630 15342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:54.702870 15342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:54.725369 15348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:54.725579 15351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:54.725883 15349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:54.726094 15342 server_base.cc:1034] running on GCE node
I20260619 14:52:54.727033 15342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:54.728652 15342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:54.729916 15342 hybrid_clock.cc:648] HybridClock initialized: now 1781880774729857 us; error 61 us; skew 500 ppm
I20260619 14:52:54.730346 15342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:54 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:54.734017 15342 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:52:54.735790 15342 webserver.cc:460] Webserver started at http://127.0.241.62:39741/ using document root <none> and password file <none>
I20260619 14:52:54.736639 15342 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:54.736804 15342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:54.742807 15342 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:52:54.745956 15358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:54.747344 15342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:52:54.747622 15342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "15aefc63f510414c9c7361697cb7b778"
format_stamp: "Formatted at 2026-06-19 14:52:51 on dist-test-slave-p3ml"
server_key: "bf9345d004627addda117a26d6840eb8"
server_key_iv: "bff440e7dfb65d39434677ee00af2f30"
server_key_version: "encryptionkey@0"
I20260619 14:52:54.748693 15342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:54.769230 15342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:54.773303 15342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:54.773610 15342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:54.792676 15342 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:45075
I20260619 14:52:54.792695 15409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:45075 every 8 connection(s)
I20260619 14:52:54.794214 15342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:54.797880 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 15342
I20260619 14:52:54.798846 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:45447
--local_ip_for_outbound_sockets=127.0.241.1
--tserver_master_addrs=127.0.241.62:45075
--webserver_port=46303
--webserver_interface=127.0.241.1
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:52:54.802853 15410 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:52:54.808118 15410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Bootstrap starting.
I20260619 14:52:54.827804 15410 log.cc:824] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:54.839529 15410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:52:54.840076 15410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Bootstrap complete.
I20260619 14:52:54.850651 15410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } }
I20260619 14:52:54.851476 15410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15aefc63f510414c9c7361697cb7b778, State: Initialized, Role: FOLLOWER
I20260619 14:52:54.852156 15410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } }
I20260619 14:52:54.852445 15410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:54.852563 15410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:54.852752 15410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:52:54.856668 15410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } }
I20260619 14:52:54.857218 15410 leader_election.cc:302] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15aefc63f510414c9c7361697cb7b778; no voters:
I20260619 14:52:54.857831 15410 leader_election.cc:288] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:52:54.857964 15414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:52:54.859112 15414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 2 LEADER]: Becoming Leader. State: Replica: 15aefc63f510414c9c7361697cb7b778, State: Running, Role: LEADER
I20260619 14:52:54.859750 15414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } }
I20260619 14:52:54.860175 15410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:54.862236 15416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15aefc63f510414c9c7361697cb7b778. Latest consensus state: current_term: 2 leader_uuid: "15aefc63f510414c9c7361697cb7b778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } } }
I20260619 14:52:54.862663 15416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:54.863261 15415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "15aefc63f510414c9c7361697cb7b778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } } }
I20260619 14:52:54.863605 15415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:54.870168 15420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:54.879449 15420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:54.881295 15420 catalog_manager.cc:1259] Loaded cluster ID: a1adaba70d7c4562b16f2e4c630ae838
I20260619 14:52:54.881482 15420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:54.885489 15420 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 14:52:54.915167 15420 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 14:52:54.936062 15420 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:54.937960 15420 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Loaded TSK: 0
I20260619 14:52:54.939590 15420 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:52:55.156526 15412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:55.156792 15412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:55.156883 15412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:55.164434 15412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:55.164609 15412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:55.164786 15412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:52:55.174399 15412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:45447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=46303
--tserver_master_addrs=127.0.241.62:45075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:55.175732 15412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:55.176997 15412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:55.191776 15438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:55.191799 15439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:55.192677 15441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:55.192996 15412 server_base.cc:1034] running on GCE node
I20260619 14:52:55.193697 15412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:55.194648 15412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:55.195868 15412 hybrid_clock.cc:648] HybridClock initialized: now 1781880775195805 us; error 57 us; skew 500 ppm
I20260619 14:52:55.196184 15412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:55 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:55.203734 15412 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:52:55.205425 15412 webserver.cc:460] Webserver started at http://127.0.241.1:46303/ using document root <none> and password file <none>
I20260619 14:52:55.206130 15412 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:55.206288 15412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:55.212009 15412 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:52:55.214809 15448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:55.216205 15412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:52:55.216336 15412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "4f5646b30e734d74aa51a90b5205eeec"
format_stamp: "Formatted at 2026-06-19 14:52:52 on dist-test-slave-p3ml"
server_key: "9d6635aeb47604ccba69111b7c87981b"
server_key_iv: "ac5647daa96685c47e284ef9dfb19a10"
server_key_version: "encryptionkey@0"
I20260619 14:52:55.217255 15412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:55.243805 15412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:55.248013 15412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:55.248245 15412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:55.249161 15412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:55.250396 15412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:55.250490 15412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:55.250569 15412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:55.250623 15412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:55.284300 15412 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:45447
I20260619 14:52:55.284350 15560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:45447 every 8 connection(s)
I20260619 14:52:55.285761 15412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:55.294757 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 15412
I20260619 14:52:55.295390 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:40675
--local_ip_for_outbound_sockets=127.0.241.2
--tserver_master_addrs=127.0.241.62:45075
--webserver_port=41263
--webserver_interface=127.0.241.2
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:52:55 dist-test-slave-p3ml krb5kdc[14790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:55.312067 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:55.289003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:40553)
0619 14:52:55.289887 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:55.289937 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:55.291747 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:55.293021 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:55.293028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:55.293527 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:55.294070 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:55.294087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:55.296154 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:55.296162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:55.298351 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:55.298369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:55.299421 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:55.299459 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:55.299800 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:55.300542 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:55.300583 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:55.303303 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:55.308067 (+ 4764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:55.308074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:55.308084 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:55.308489 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:55.308966 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:55.308969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:55.308972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:55.309140 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:55.310101 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:55.310106 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:55.310391 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 14:52:55.310627 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 14:52:55.311004 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":370,"threads_started":1}
I20260619 14:52:55.312561 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:55.288785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:40553 (local address 127.0.241.62:45075)
0619 14:52:55.289324 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:55.289336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:55.290066 (+ 730us) server_negotiation.cc:407] Connection header received
0619 14:52:55.292245 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:55.292258 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:55.292598 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:55.292886 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:55.294332 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:55.295982 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:55.298536 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:55.299181 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:55.299314 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:55.303666 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:55.303680 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:55.303702 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:55.303762 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:55.307930 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:55.308624 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:55.308636 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:55.308647 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:55.308847 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:55.309248 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:55.309255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:55.309261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:55.309954 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:55.310165 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:55.310882 (+ 717us) server_negotiation.cc:299] Negotiation successful
0619 14:52:55.311333 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":229,"threads_started":1}
I20260619 14:52:55.315464 15561 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45075
I20260619 14:52:55.315829 15561 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:55.316767 15561 heartbeater.cc:511] Master 127.0.241.62:45075 requested a full tablet report, sending...
I20260619 14:52:55.319168 15375 ts_manager.cc:194] Registered new tserver with Master: 4f5646b30e734d74aa51a90b5205eeec (127.0.241.1:45447)
I20260619 14:52:55.321097 15375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:40553
W20260619 14:52:55.629197 15565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:55.629492 15565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:55.629587 15565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:55.636864 15565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:55.636984 15565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:55.637099 15565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:52:55.646873 15565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:40675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=41263
--tserver_master_addrs=127.0.241.62:45075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:55.648169 15565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:55.649581 15565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:55.663098 15570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:55.663020 15573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:55.663110 15571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:55.663723 15565 server_base.cc:1034] running on GCE node
I20260619 14:52:55.664453 15565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:55.665298 15565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:55.666524 15565 hybrid_clock.cc:648] HybridClock initialized: now 1781880775666483 us; error 51 us; skew 500 ppm
I20260619 14:52:55.666844 15565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:55 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:55.669916 15565 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:52:55.671345 15565 webserver.cc:460] Webserver started at http://127.0.241.2:41263/ using document root <none> and password file <none>
I20260619 14:52:55.672080 15565 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:55.672245 15565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:55.677490 15565 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:52:55.680126 15580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:55.681521 15565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:52:55.681696 15565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1"
format_stamp: "Formatted at 2026-06-19 14:52:52 on dist-test-slave-p3ml"
server_key: "55a996edc14d7ce0f0f685b0eefb9d5a"
server_key_iv: "4d39f1e225c6444b53811e2e2cf62c7f"
server_key_version: "encryptionkey@0"
I20260619 14:52:55.682724 15565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:55.700810 15565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:55.705053 15565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:55.705344 15565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:55.706337 15565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:55.707783 15565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:55.707948 15565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:55.708076 15565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:55.708168 15565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:55.741508 15565 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:40675
I20260619 14:52:55.741556 15692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:40675 every 8 connection(s)
I20260619 14:52:55.742949 15565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:55.752889 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 15565
I20260619 14:52:55.753542 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:38665
--local_ip_for_outbound_sockets=127.0.241.3
--tserver_master_addrs=127.0.241.62:45075
--webserver_port=43225
--webserver_interface=127.0.241.3
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:52:55 dist-test-slave-p3ml krb5kdc[14790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:55.762738 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:55.745450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:54745 (local address 127.0.241.62:45075)
0619 14:52:55.745682 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:55.745689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:55.746786 (+ 1097us) server_negotiation.cc:407] Connection header received
0619 14:52:55.748694 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:55.748698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:55.748764 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:55.748849 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:55.750249 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:55.750932 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:55.752557 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:55.753056 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:55.753194 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:55.756897 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:55.756904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:55.756909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:55.756948 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:55.759512 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:55.760324 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:55.760330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:55.760334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:55.760466 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:55.761008 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:55.761011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:55.761013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:55.761309 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:55.761464 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:55.762154 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 14:52:55.762389 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 14:52:55.763787 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:55.745963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:54745)
0619 14:52:55.746630 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:55.746665 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:55.748454 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:55.749054 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:55.749061 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:55.749560 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:55.750107 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:55.750124 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:55.751063 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:55.751066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:55.752360 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:55.752372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:55.753311 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:55.753319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:55.753561 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:55.754313 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:55.754336 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:55.756692 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:55.759647 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:55.759656 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:55.759672 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:55.760198 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:55.760603 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:55.760609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:55.760613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:55.760855 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:55.761434 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:55.761442 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:55.761792 (+ 350us) client_negotiation.cc:769] Sending connection context
0619 14:52:55.762059 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 14:52:55.762567 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":264,"threads_started":1}
I20260619 14:52:55.765254 15693 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45075
I20260619 14:52:55.765592 15693 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:55.766453 15693 heartbeater.cc:511] Master 127.0.241.62:45075 requested a full tablet report, sending...
I20260619 14:52:55.768149 15375 ts_manager.cc:194] Registered new tserver with Master: dccd7fd581cb43aa968a4e8f8ad0b4f1 (127.0.241.2:40675)
I20260619 14:52:55.769528 15375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:54745
W20260619 14:52:56.082187 15696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:56.082460 15696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:56.082553 15696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:56.089901 15696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:56.090008 15696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:56.090124 15696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:52:56.099644 15696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:38665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=43225
--tserver_master_addrs=127.0.241.62:45075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:56.100849 15696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:56.102065 15696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:56.115047 15701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:56.115088 15702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:56.115088 15704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:56.115792 15696 server_base.cc:1034] running on GCE node
I20260619 14:52:56.116376 15696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:56.117293 15696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:56.118543 15696 hybrid_clock.cc:648] HybridClock initialized: now 1781880776118439 us; error 85 us; skew 500 ppm
I20260619 14:52:56.118853 15696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:56 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:56.122363 15696 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:52:56.123744 15696 webserver.cc:460] Webserver started at http://127.0.241.3:43225/ using document root <none> and password file <none>
I20260619 14:52:56.124361 15696 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:56.124469 15696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:56.129583 15696 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 14:52:56.132055 15711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:56.133363 15696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:52:56.133538 15696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "41da6d08489742bbbc2a5cc12946e6a7"
format_stamp: "Formatted at 2026-06-19 14:52:53 on dist-test-slave-p3ml"
server_key: "260a942338b6ebc56f7b00c1cc004515"
server_key_iv: "e6e3cd7f38ccc8581a41cc03c8c2cb14"
server_key_version: "encryptionkey@0"
I20260619 14:52:56.134491 15696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:56.152757 15696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:56.157208 15696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:56.157495 15696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:56.158483 15696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:56.159782 15696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:56.159907 15696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:56.160046 15696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:56.160135 15696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:56.193369 15696 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:38665
I20260619 14:52:56.193457 15823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:38665 every 8 connection(s)
I20260619 14:52:56.194816 15696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:56.197783 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 15696
Jun 19 14:52:56 dist-test-slave-p3ml krb5kdc[14790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:52:56.213469 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.198796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.0.241.62:45075)
0619 14:52:56.199711 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:56.199719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:56.199970 (+ 251us) server_negotiation.cc:407] Connection header received
0619 14:52:56.200179 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:56.200188 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:56.200251 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:56.200363 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:56.201853 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.202616 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.204033 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.204465 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.204610 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.207378 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:56.207384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:56.207387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:56.207418 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:56.210661 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:56.211360 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:56.211369 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:56.211372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:56.211564 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:56.211978 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:56.211983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:56.211987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:56.212273 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:56.212531 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:56.212831 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 14:52:56.213050 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":638,"threads_started":1}
I20260619 14:52:56.215323 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.197371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:47195 (local address 127.0.241.62:45075)
0619 14:52:56.197551 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:56.197555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:56.199091 (+ 1536us) server_negotiation.cc:407] Connection header received
0619 14:52:56.201262 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:56.201267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:56.201354 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:56.201464 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:56.202971 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.203769 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.205289 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.205723 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.205858 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.209414 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:56.209419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:56.209422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:56.209466 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:56.212277 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:56.213013 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:56.213018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:56.213020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:56.213129 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:56.213696 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:56.213699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:56.213701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:56.213935 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:56.214100 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:56.214773 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 14:52:56.214973 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:52:56.216400 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.197972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:47195)
0619 14:52:56.198907 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:56.198958 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:56.200964 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:56.201727 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:56.201735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:56.202213 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:56.202804 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:56.202820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.203922 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.203934 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:56.205141 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:56.205152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.205950 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.205957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:56.206198 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.206838 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:56.206884 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:56.209198 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:56.212422 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:56.212431 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:56.212449 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:56.212868 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:56.213266 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:56.213271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:56.213275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:56.213461 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:56.214077 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:56.214084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:56.214423 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 14:52:56.214697 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 14:52:56.215207 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":355,"threads_started":1}
I20260619 14:52:56.217648 15824 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45075
I20260619 14:52:56.217933 15824 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:56.218678 15824 heartbeater.cc:511] Master 127.0.241.62:45075 requested a full tablet report, sending...
I20260619 14:52:56.220016 15375 ts_manager.cc:194] Registered new tserver with Master: 41da6d08489742bbbc2a5cc12946e6a7 (127.0.241.3:38665)
I20260619 14:52:56.220927 15375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:47195
I20260619 14:52:56.228289 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:52:56.234946 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.229137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.0.241.62:45075)
0619 14:52:56.229352 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:56.229361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:56.229689 (+ 328us) server_negotiation.cc:407] Connection header received
0619 14:52:56.229852 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:56.229855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:56.229978 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:56.230115 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:56.230487 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.231224 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.232757 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.233179 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.233286 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.233736 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:56.234317 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:56.234474 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:56.234607 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 14:52:56.234685 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:52:56.235903 15375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:52:56.243404 15375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:52:56.266942 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.255815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.0.241.1:45447)
0619 14:52:56.257027 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:56.257040 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:56.257972 (+ 932us) server_negotiation.cc:407] Connection header received
0619 14:52:56.258277 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:56.258287 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:56.258546 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:56.258751 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:56.260035 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.261735 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.264367 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.265709 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.265842 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.266042 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:56.266382 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 14:52:56.266605 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":851,"threads_started":1}
I20260619 14:52:56.266981 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.255459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45447 (local address 127.0.0.1:49412)
0619 14:52:56.257494 (+ 2035us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:56.257510 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:56.258070 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:56.259028 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:56.259036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:56.259064 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:56.259733 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:56.259751 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.261872 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.261877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:56.264158 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:56.264171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.265912 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.265919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:56.266115 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.266171 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 14:52:56.266295 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 14:52:56.266494 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1912,"mutex_wait_us":87,"thread_start_us":1746,"threads_started":1}
I20260619 14:52:56.267554 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.254907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:38665 (local address 127.0.0.1:55486)
0619 14:52:56.256050 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:56.256104 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:56.256303 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:56.259223 (+ 2920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:56.259231 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:56.259258 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:56.259847 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:56.259857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.262290 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.262298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:56.264874 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:56.264889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.266670 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.266677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:56.266857 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.266890 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:52:56.266963 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:52:56.267132 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":125,"thread_start_us":428,"threads_started":1}
I20260619 14:52:56.268102 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.256248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:40675 (local address 127.0.0.1:56398)
0619 14:52:56.259523 (+ 3275us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:56.259550 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:56.259769 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:56.260647 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:56.260651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:56.260674 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:56.261301 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:56.261312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.262982 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.262987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:56.265223 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:56.265233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.267171 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.267178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:56.267387 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.267479 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 14:52:56.267595 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 14:52:56.267716 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3113,"mutex_wait_us":59,"thread_start_us":2931,"threads_started":1}
I20260619 14:52:56.268745 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.255118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.0.241.3:38665)
0619 14:52:56.258639 (+ 3521us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:56.258649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:56.258681 (+ 32us) server_negotiation.cc:407] Connection header received
0619 14:52:56.258777 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:56.258784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:56.258989 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:56.259393 (+ 404us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:56.260760 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.262157 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.265058 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.266377 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.267566 (+ 1189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.267796 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:56.267883 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 14:52:56.268268 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3351,"thread_start_us":3200,"threads_started":1}
I20260619 14:52:56.268841 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.256891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.0.241.2:40675)
0619 14:52:56.258944 (+ 2053us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:56.258956 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:56.259871 (+ 915us) server_negotiation.cc:407] Connection header received
0619 14:52:56.260053 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:56.260069 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:56.260298 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:56.260481 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:56.261449 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.262836 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.265357 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.266656 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.266775 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.266972 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:56.268351 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0619 14:52:56.268519 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1875,"thread_start_us":1756,"threads_started":1}
I20260619 14:52:56.271204 15496 tablet_service.cc:1467] Processing CreateTablet for tablet ba2e597b73724956821ccbddee526f33 (DEFAULT_TABLE table=test-table [id=51e4f5ca61624906819b20d841524bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:56.272228 15759 tablet_service.cc:1467] Processing CreateTablet for tablet ba2e597b73724956821ccbddee526f33 (DEFAULT_TABLE table=test-table [id=51e4f5ca61624906819b20d841524bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:56.272982 15496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2e597b73724956821ccbddee526f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:56.272796 15628 tablet_service.cc:1467] Processing CreateTablet for tablet ba2e597b73724956821ccbddee526f33 (DEFAULT_TABLE table=test-table [id=51e4f5ca61624906819b20d841524bba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:56.273588 15759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2e597b73724956821ccbddee526f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:56.273993 15628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba2e597b73724956821ccbddee526f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:56.292409 15840 tablet_bootstrap.cc:492] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec: Bootstrap starting.
I20260619 14:52:56.296284 15840 tablet_bootstrap.cc:654] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:56.297631 15840 log.cc:824] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:56.302291 15841 tablet_bootstrap.cc:492] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Bootstrap starting.
I20260619 14:52:56.302611 15840 tablet_bootstrap.cc:492] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec: No bootstrap required, opened a new log
I20260619 14:52:56.302903 15840 ts_tablet_manager.cc:1397] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260619 14:52:56.303709 15842 tablet_bootstrap.cc:492] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7: Bootstrap starting.
I20260619 14:52:56.306118 15841 tablet_bootstrap.cc:654] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:56.306968 15842 tablet_bootstrap.cc:654] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:56.307257 15841 log.cc:824] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:56.310386 15841 tablet_bootstrap.cc:492] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1: No bootstrap required, opened a new log
I20260619 14:52:56.310647 15841 ts_tablet_manager.cc:1397] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 14:52:56.311157 15842 log.cc:824] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:56.313680 15842 tablet_bootstrap.cc:492] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7: No bootstrap required, opened a new log
I20260619 14:52:56.313576 15840 raft_consensus.cc:348] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:56.313939 15842 ts_tablet_manager.cc:1397] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260619 14:52:56.314064 15840 raft_consensus.cc:374] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:56.314172 15840 raft_consensus.cc:729] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5646b30e734d74aa51a90b5205eeec, State: Initialized, Role: FOLLOWER
I20260619 14:52:56.314831 15840 consensus_queue.cc:260] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:56.316037 15561 heartbeater.cc:503] Master 127.0.241.62:45075 was elected leader, sending a full tablet report...
I20260619 14:52:56.316753 15840 ts_tablet_manager.cc:1428] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260619 14:52:56.321550 15841 raft_consensus.cc:348] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:56.321940 15841 raft_consensus.cc:374] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:56.322077 15841 raft_consensus.cc:729] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccd7fd581cb43aa968a4e8f8ad0b4f1, State: Initialized, Role: FOLLOWER
I20260619 14:52:56.322865 15841 consensus_queue.cc:260] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:56.323014 15842 raft_consensus.cc:348] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:56.323307 15842 raft_consensus.cc:374] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:56.323406 15842 raft_consensus.cc:729] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41da6d08489742bbbc2a5cc12946e6a7, State: Initialized, Role: FOLLOWER
I20260619 14:52:56.324004 15842 consensus_queue.cc:260] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:56.324108 15693 heartbeater.cc:503] Master 127.0.241.62:45075 was elected leader, sending a full tablet report...
I20260619 14:52:56.324800 15841 ts_tablet_manager.cc:1428] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260619 14:52:56.325263 15824 heartbeater.cc:503] Master 127.0.241.62:45075 was elected leader, sending a full tablet report...
I20260619 14:52:56.325691 15842 ts_tablet_manager.cc:1428] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260619 14:52:56.416667 15846 raft_consensus.cc:482] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:56.416924 15846 raft_consensus.cc:504] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:56.418465 15846 leader_election.cc:288] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41da6d08489742bbbc2a5cc12946e6a7 (127.0.241.3:38665), dccd7fd581cb43aa968a4e8f8ad0b4f1 (127.0.241.2:40675)
I20260619 14:52:56.425411 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.419779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:35737 (local address 127.0.241.2:40675)
0619 14:52:56.419990 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:56.419996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:56.420232 (+ 236us) server_negotiation.cc:407] Connection header received
0619 14:52:56.420438 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:56.420442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:56.420496 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:56.420601 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:56.421106 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.421792 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.423670 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.424407 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.424572 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.424674 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:56.425016 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 14:52:56.425139 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 14:52:56.425468 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.418825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:35321 (local address 127.0.241.3:38665)
0619 14:52:56.419072 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:56.419079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:56.419493 (+ 414us) server_negotiation.cc:407] Connection header received
0619 14:52:56.419754 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:56.419763 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:56.419861 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:56.419958 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:56.420750 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.421640 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.423557 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.424377 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.424475 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.424613 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:56.424968 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 14:52:56.425095 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:52:56.425518 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.419521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:40675 (local address 127.0.241.1:35737)
0619 14:52:56.420119 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:56.420139 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:56.420296 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:56.420606 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:56.420610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:56.420632 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:56.420955 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:56.420963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.421927 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.421931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:56.423523 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:56.423538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.424676 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.424682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:56.424840 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.424861 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:52:56.424974 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 14:52:56.425101 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":65,"thread_start_us":285,"threads_started":1}
I20260619 14:52:56.425773 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.418763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:38665 (local address 127.0.241.1:35321)
0619 14:52:56.419345 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:56.419361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:56.419572 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:56.420031 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:56.420035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:56.420063 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:56.420510 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:56.420520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.421816 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.421822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:56.423394 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:56.423406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.424587 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.424593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:56.424782 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.424833 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 14:52:56.424904 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:52:56.425033 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":49,"spinlock_wait_cycles":15104,"thread_start_us":290,"threads_started":1}
I20260619 14:52:56.426501 15648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2e597b73724956821ccbddee526f33" candidate_uuid: "4f5646b30e734d74aa51a90b5205eeec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" is_pre_election: true
I20260619 14:52:56.426621 15779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2e597b73724956821ccbddee526f33" candidate_uuid: "4f5646b30e734d74aa51a90b5205eeec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41da6d08489742bbbc2a5cc12946e6a7" is_pre_election: true
I20260619 14:52:56.427014 15648 raft_consensus.cc:2454] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5646b30e734d74aa51a90b5205eeec in term 0.
I20260619 14:52:56.427107 15779 raft_consensus.cc:2454] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5646b30e734d74aa51a90b5205eeec in term 0.
I20260619 14:52:56.427729 15449 leader_election.cc:302] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41da6d08489742bbbc2a5cc12946e6a7, 4f5646b30e734d74aa51a90b5205eeec; no voters:
I20260619 14:52:56.428092 15846 raft_consensus.cc:2781] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:56.428248 15846 raft_consensus.cc:482] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:56.428355 15846 raft_consensus.cc:3037] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:56.430488 15846 raft_consensus.cc:504] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:56.431020 15846 leader_election.cc:288] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [CANDIDATE]: Term 1 election: Requested vote from peers 41da6d08489742bbbc2a5cc12946e6a7 (127.0.241.3:38665), dccd7fd581cb43aa968a4e8f8ad0b4f1 (127.0.241.2:40675)
I20260619 14:52:56.431510 15779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2e597b73724956821ccbddee526f33" candidate_uuid: "4f5646b30e734d74aa51a90b5205eeec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41da6d08489742bbbc2a5cc12946e6a7"
I20260619 14:52:56.431609 15648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba2e597b73724956821ccbddee526f33" candidate_uuid: "4f5646b30e734d74aa51a90b5205eeec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1"
I20260619 14:52:56.431694 15779 raft_consensus.cc:3037] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:56.431833 15648 raft_consensus.cc:3037] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:56.433914 15779 raft_consensus.cc:2454] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5646b30e734d74aa51a90b5205eeec in term 1.
I20260619 14:52:56.434020 15648 raft_consensus.cc:2454] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5646b30e734d74aa51a90b5205eeec in term 1.
I20260619 14:52:56.434336 15449 leader_election.cc:302] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41da6d08489742bbbc2a5cc12946e6a7, 4f5646b30e734d74aa51a90b5205eeec; no voters:
I20260619 14:52:56.434643 15846 raft_consensus.cc:2781] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:56.435114 15846 raft_consensus.cc:686] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 1 LEADER]: Becoming Leader. State: Replica: 4f5646b30e734d74aa51a90b5205eeec, State: Running, Role: LEADER
I20260619 14:52:56.435678 15846 consensus_queue.cc:237] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } }
I20260619 14:52:56.440413 15374 catalog_manager.cc:5495] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f5646b30e734d74aa51a90b5205eeec (127.0.241.1). New cstate: current_term: 1 leader_uuid: "4f5646b30e734d74aa51a90b5205eeec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5646b30e734d74aa51a90b5205eeec" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45447 } health_report { overall_health: HEALTHY } } }
W20260619 14:52:56.446471 15825 tablet.cc:2369] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:56.458163 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.453344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.0.241.1:45447)
0619 14:52:56.453534 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:56.453538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:56.453574 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:52:56.453744 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:56.453748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:56.453821 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:56.453936 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:56.454275 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.454935 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.456298 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.456692 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.456784 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.457182 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:56.457623 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:56.457724 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:56.457815 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 14:52:56.457876 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:52:56.465866 15779 raft_consensus.cc:1261] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7 [term 1 FOLLOWER]: Refusing update from remote peer 4f5646b30e734d74aa51a90b5205eeec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:56.465941 15648 raft_consensus.cc:1261] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 1 FOLLOWER]: Refusing update from remote peer 4f5646b30e734d74aa51a90b5205eeec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:56.466850 15851 consensus_queue.cc:1035] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [LEADER]: Connected to new peer: Peer: permanent_uuid: "41da6d08489742bbbc2a5cc12946e6a7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:56.467218 15846 consensus_queue.cc:1035] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [LEADER]: Connected to new peer: Peer: permanent_uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 40675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:56.478204 15853 mvcc.cc:205] Tried to move back new op lower bound from 7298583660395110400 to 7298583660284645376. Current Snapshot: MvccSnapshot[applied={T|T < 7298583660395110400}]
I20260619 14:52:56.491362 15374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:56.491688 15374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:56.495479 15374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Sending DeleteTablet for 3 replicas of tablet ba2e597b73724956821ccbddee526f33
I20260619 14:52:56.496681 15759 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2e597b73724956821ccbddee526f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:56 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:55486
I20260619 14:52:56.496693 15628 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2e597b73724956821ccbddee526f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:56 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:56398
I20260619 14:52:56.496960 15496 tablet_service.cc:1514] Processing DeleteTablet for tablet ba2e597b73724956821ccbddee526f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:56 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:49412
I20260619 14:52:56.497816 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 15412
I20260619 14:52:56.512928 15865 tablet_replica.cc:330] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec: stopping tablet replica
I20260619 14:52:56.513525 15865 raft_consensus.cc:2229] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:52:56.514308 15865 raft_consensus.cc:2258] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:56.517062 15865 ts_tablet_manager.cc:1905] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:56.527132 15865 ts_tablet_manager.cc:1918] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:56.527350 15865 log.cc:1196] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/ba2e597b73724956821ccbddee526f33
I20260619 14:52:56.528030 15865 ts_tablet_manager.cc:1939] T ba2e597b73724956821ccbddee526f33 P 4f5646b30e734d74aa51a90b5205eeec: Deleting consensus metadata
I20260619 14:52:56.529208 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.499588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.0.241.1:45447)
0619 14:52:56.499800 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:56.499808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:56.499830 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:52:56.499904 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:56.499912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:56.500012 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:56.500131 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:56.502692 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.504385 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.507119 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.508157 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.508313 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.517596 (+ 9283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:56.517608 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:56.517632 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:56.517683 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:56.522322 (+ 4639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:56.525658 (+ 3336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:56.525672 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:56.525678 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:56.525900 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:56.526336 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:56.526344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:56.526348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:56.526937 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:56.527281 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:56.528355 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0619 14:52:56.528672 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:52:56.529837 15863 tablet_replica.cc:330] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1: stopping tablet replica
I20260619 14:52:56.529994 15556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:56.530272 15864 tablet_replica.cc:330] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7: stopping tablet replica
I20260619 14:52:56.530320 15863 raft_consensus.cc:2229] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:56.530869 15863 raft_consensus.cc:2258] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:56.531131 15361 catalog_manager.cc:4841] TS 4f5646b30e734d74aa51a90b5205eeec (127.0.241.1:45447): tablet ba2e597b73724956821ccbddee526f33 (table test-table [id=51e4f5ca61624906819b20d841524bba]) successfully deleted
I20260619 14:52:56.531527 15864 raft_consensus.cc:2229] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:56.532027 15864 raft_consensus.cc:2258] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:56.533172 15863 ts_tablet_manager.cc:1905] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:56.534457 15864 ts_tablet_manager.cc:1905] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:56.539909 15864 ts_tablet_manager.cc:1918] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:56.540094 15864 log.cc:1196] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/ba2e597b73724956821ccbddee526f33
I20260619 14:52:56.540544 15864 ts_tablet_manager.cc:1939] T ba2e597b73724956821ccbddee526f33 P 41da6d08489742bbbc2a5cc12946e6a7: Deleting consensus metadata
I20260619 14:52:56.541256 15863 ts_tablet_manager.cc:1918] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:56.541428 15863 log.cc:1196] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/ba2e597b73724956821ccbddee526f33
I20260619 14:52:56.541972 15863 ts_tablet_manager.cc:1939] T ba2e597b73724956821ccbddee526f33 P dccd7fd581cb43aa968a4e8f8ad0b4f1: Deleting consensus metadata
I20260619 14:52:56.542181 15359 catalog_manager.cc:4841] TS 41da6d08489742bbbc2a5cc12946e6a7 (127.0.241.3:38665): tablet ba2e597b73724956821ccbddee526f33 (table test-table [id=51e4f5ca61624906819b20d841524bba]) successfully deleted
I20260619 14:52:56.543788 15359 catalog_manager.cc:4841] TS dccd7fd581cb43aa968a4e8f8ad0b4f1 (127.0.241.2:40675): tablet ba2e597b73724956821ccbddee526f33 (table test-table [id=51e4f5ca61624906819b20d841524bba]) successfully deleted
I20260619 14:52:56.630734 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 15412
I20260619 14:52:56.649240 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 15565
I20260619 14:52:56.663225 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.650027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.0.241.2:40675)
0619 14:52:56.650233 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:56.650241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:56.650261 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:52:56.650507 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:56.650512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:56.650585 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:56.650700 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:56.652171 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.652899 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.654189 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.654592 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.654732 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.657436 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:56.657449 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:56.657470 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:56.657504 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:56.660475 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:56.661221 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:56.661228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:56.661230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:56.661351 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:56.661729 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:56.661736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:56.661739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:56.662076 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:56.662360 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:56.662673 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 14:52:56.662883 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 14:52:56.663902 15688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:56.752435 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 15565
I20260619 14:52:56.769522 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 15696
I20260619 14:52:56.782124 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:56.770288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.0.241.3:38665)
0619 14:52:56.770447 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:56.770452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:56.770470 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:52:56.770611 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:56.770615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:56.770692 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:56.770777 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:56.772249 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.773018 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.774371 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:56.774753 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:56.774841 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:56.777272 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:56.777284 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:56.777299 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:56.777327 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:56.779860 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:56.780498 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:56.780504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:56.780506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:56.780648 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:56.780953 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:56.780957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:56.780958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:56.781286 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:56.781488 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:56.781706 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 14:52:56.781867 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:52:56.782742 15819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:56.870864 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 15696
I20260619 14:52:56.887692 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 15342
I20260619 14:52:56.888875 15405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:52:57.000505 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 15342
I20260619 14:52:57.021780 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45075
--webserver_interface=127.0.241.62
--webserver_port=39741
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:57.347368 15870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:57.347792 15870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:57.347952 15870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:57.355182 15870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:57.355319 15870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:57.355466 15870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:57.355576 15870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:57.355655 15870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:57.365144 15870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:45075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=39741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:57.366436 15870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:57.367739 15870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:57.379514 15876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:57.379472 15878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:57.379563 15875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:57.380313 15870 server_base.cc:1034] running on GCE node
I20260619 14:52:57.381080 15870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:57.382436 15870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:57.383666 15870 hybrid_clock.cc:648] HybridClock initialized: now 1781880777383614 us; error 60 us; skew 500 ppm
I20260619 14:52:57.384033 15870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:57 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:57.387017 15870 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:52:57.388550 15870 webserver.cc:460] Webserver started at http://127.0.241.62:39741/ using document root <none> and password file <none>
I20260619 14:52:57.389231 15870 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:57.389330 15870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:57.394714 15870 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 14:52:57.397514 15885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:57.398813 15870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:52:57.398952 15870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "15aefc63f510414c9c7361697cb7b778"
format_stamp: "Formatted at 2026-06-19 14:52:51 on dist-test-slave-p3ml"
server_key: "bf9345d004627addda117a26d6840eb8"
server_key_iv: "bff440e7dfb65d39434677ee00af2f30"
server_key_version: "encryptionkey@0"
I20260619 14:52:57.399984 15870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:57.417316 15870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:57.421062 15870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:57.421280 15870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:57.439993 15870 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:45075
I20260619 14:52:57.440012 15936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:45075 every 8 connection(s)
I20260619 14:52:57.441560 15870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:57.444185 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 15870
I20260619 14:52:57.445147 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:45447
--local_ip_for_outbound_sockets=127.0.241.1
--tserver_master_addrs=127.0.241.62:45075
--webserver_port=46303
--webserver_interface=127.0.241.1
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:52:57.451067 15937 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:52:57.455343 15937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Bootstrap starting.
I20260619 14:52:57.475983 15937 log.cc:824] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:57.491942 15937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:52:57.492493 15937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778: Bootstrap complete.
I20260619 14:52:57.501662 15937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } }
I20260619 14:52:57.502524 15937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15aefc63f510414c9c7361697cb7b778, State: Initialized, Role: FOLLOWER
I20260619 14:52:57.503211 15937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } }
I20260619 14:52:57.503474 15937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:57.503592 15937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:57.503789 15937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:52:57.507648 15937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } }
I20260619 14:52:57.508128 15937 leader_election.cc:302] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15aefc63f510414c9c7361697cb7b778; no voters:
I20260619 14:52:57.508870 15937 leader_election.cc:288] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:52:57.508999 15941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:52:57.511211 15937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:57.511721 15941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [term 3 LEADER]: Becoming Leader. State: Replica: 15aefc63f510414c9c7361697cb7b778, State: Running, Role: LEADER
I20260619 14:52:57.512238 15941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } }
I20260619 14:52:57.515059 15942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "15aefc63f510414c9c7361697cb7b778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } } }
I20260619 14:52:57.515421 15942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:57.515825 15943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15aefc63f510414c9c7361697cb7b778. Latest consensus state: current_term: 3 leader_uuid: "15aefc63f510414c9c7361697cb7b778" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aefc63f510414c9c7361697cb7b778" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45075 } } }
I20260619 14:52:57.516273 15943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aefc63f510414c9c7361697cb7b778 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:57.523792 15950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:57.532543 15950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:57.534037 15950 catalog_manager.cc:1259] Loaded cluster ID: a1adaba70d7c4562b16f2e4c630ae838
I20260619 14:52:57.534134 15950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 14:52:57.536017 15950 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f52cafb19ad google::LogMessage::Fail() at ??:0
@ 0x7f52cafb488a google::LogMessage::SendToLog() at ??:0
@ 0x7f52cafb14de google::LogMessage::Flush() at ??:0
@ 0x7f52cafb2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f52e09e4512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f52e09e1976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f52e0a69610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f52e0a69439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f52cdbfd345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f52cdc0091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f52cdc00799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f52cdbdbeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f52d0eb56db start_thread at ??:0
@ 0x7f52c9e4b61f clone at ??:0
W20260619 14:52:57.846308 15939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:57.846591 15939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:57.846689 15939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:57.854215 15939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:57.854404 15939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:57.854636 15939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:52:57.865429 15939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:45447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=46303
--tserver_master_addrs=127.0.241.62:45075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:57.866703 15939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:57.868039 15939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:57.881086 15978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:57.881160 15981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:57.881158 15979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:57.881705 15939 server_base.cc:1034] running on GCE node
I20260619 14:52:57.882292 15939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:57.883198 15939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:57.884426 15939 hybrid_clock.cc:648] HybridClock initialized: now 1781880777884359 us; error 81 us; skew 500 ppm
I20260619 14:52:57.884774 15939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:57 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:57.887907 15939 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:52:57.889214 15939 webserver.cc:460] Webserver started at http://127.0.241.1:46303/ using document root <none> and password file <none>
I20260619 14:52:57.889837 15939 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:57.889930 15939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:57.895010 15939 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:52:57.897631 15988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:57.898910 15939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:52:57.899044 15939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "4f5646b30e734d74aa51a90b5205eeec"
format_stamp: "Formatted at 2026-06-19 14:52:52 on dist-test-slave-p3ml"
server_key: "9d6635aeb47604ccba69111b7c87981b"
server_key_iv: "ac5647daa96685c47e284ef9dfb19a10"
server_key_version: "encryptionkey@0"
I20260619 14:52:57.900053 15939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:57.920632 15939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:57.924647 15939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:57.924857 15939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:57.925733 15939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:57.926949 15939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:57.927033 15939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:57.927121 15939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:57.927176 15939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:57.960911 15939 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:45447
I20260619 14:52:57.961006 16100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:45447 every 8 connection(s)
I20260619 14:52:57.962441 15939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:57.966931 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:57.965173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:51765)
0619 14:52:57.965883 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:57.966084 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":285,"threads_started":1}
W20260619 14:52:57.967656 16101 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:45075 (0 consecutive failures): Network error: Failed to ping master at 127.0.241.62:45075: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
I20260619 14:52:57.968417 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:57.968129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:52683)
0619 14:52:57.968255 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:57.968327 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 14:52:57.969177 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:57.968923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:53717)
0619 14:52:57.969023 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:57.969082 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:52:57.969429 16101 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:52:57.972762 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 15939
I20260619 14:52:57.973346 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:40675
--local_ip_for_outbound_sockets=127.0.241.2
--tserver_master_addrs=127.0.241.62:45075
--webserver_port=41263
--webserver_interface=127.0.241.2
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:58.295997 16104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:58.296265 16104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:58.296464 16104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:58.303829 16104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:58.303962 16104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:58.304183 16104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:52:58.314083 16104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:40675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=41263
--tserver_master_addrs=127.0.241.62:45075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:58.315403 16104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:58.316766 16104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:58.328783 16109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:58.329979 16110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:58.330178 16112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:58.331104 16104 server_base.cc:1034] running on GCE node
I20260619 14:52:58.331810 16104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:58.332712 16104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:58.333936 16104 hybrid_clock.cc:648] HybridClock initialized: now 1781880778333904 us; error 32 us; skew 500 ppm
I20260619 14:52:58.334270 16104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:58 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:58.337373 16104 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:52:58.338840 16104 webserver.cc:460] Webserver started at http://127.0.241.2:41263/ using document root <none> and password file <none>
I20260619 14:52:58.339583 16104 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:58.339689 16104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:58.344863 16104 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:52:58.347316 16119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:58.348603 16104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:52:58.348774 16104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "dccd7fd581cb43aa968a4e8f8ad0b4f1"
format_stamp: "Formatted at 2026-06-19 14:52:52 on dist-test-slave-p3ml"
server_key: "55a996edc14d7ce0f0f685b0eefb9d5a"
server_key_iv: "4d39f1e225c6444b53811e2e2cf62c7f"
server_key_version: "encryptionkey@0"
I20260619 14:52:58.349722 16104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:58.362178 16104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:58.366014 16104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:58.366292 16104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:58.367223 16104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:58.368546 16104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:58.368698 16104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:58.368835 16104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:58.368925 16104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:58.402545 16104 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:40675
I20260619 14:52:58.402639 16231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:40675 every 8 connection(s)
I20260619 14:52:58.404201 16104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:58.405422 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 16104
I20260619 14:52:58.406180 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:38665
--local_ip_for_outbound_sockets=127.0.241.3
--tserver_master_addrs=127.0.241.62:45075
--webserver_port=43225
--webserver_interface=127.0.241.3
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:38179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:52:58.409770 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:58.407702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:46213)
0619 14:52:58.408579 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:58.408849 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":403,"threads_started":1}
W20260619 14:52:58.410642 16232 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:45075 (0 consecutive failures): Network error: Failed to ping master at 127.0.241.62:45075: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
I20260619 14:52:58.411571 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:58.411158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:57099)
0619 14:52:58.411328 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:58.411406 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 14:52:58.412541 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:58.412096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:35639)
0619 14:52:58.412278 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:58.412383 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 14:52:58.412793 16232 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:52:58.728267 16235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:58.728519 16235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:58.728715 16235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:58.736028 16235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:58.736131 16235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:58.736333 16235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:52:58.745992 16235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:38179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:38665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=43225
--tserver_master_addrs=127.0.241.62:45075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:52:58.747314 16235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:58.748653 16235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:58.762025 16243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:58.762104 16240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:58.762104 16241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:58.762951 16235 server_base.cc:1034] running on GCE node
I20260619 14:52:58.763603 16235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:58.764467 16235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:58.765691 16235 hybrid_clock.cc:648] HybridClock initialized: now 1781880778765641 us; error 61 us; skew 500 ppm
I20260619 14:52:58.765992 16235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:52:58 dist-test-slave-p3ml krb5kdc[14790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:58.769083 16235 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:52:58.770484 16235 webserver.cc:460] Webserver started at http://127.0.241.3:43225/ using document root <none> and password file <none>
I20260619 14:52:58.771126 16235 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:58.771277 16235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:58.776268 16235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:52:58.778824 16250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:58.780094 16235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:52:58.780285 16235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "41da6d08489742bbbc2a5cc12946e6a7"
format_stamp: "Formatted at 2026-06-19 14:52:53 on dist-test-slave-p3ml"
server_key: "260a942338b6ebc56f7b00c1cc004515"
server_key_iv: "e6e3cd7f38ccc8581a41cc03c8c2cb14"
server_key_version: "encryptionkey@0"
I20260619 14:52:58.781267 16235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:58.798053 16235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:58.801878 16235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:58.802188 16235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:58.803218 16235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:58.804692 16235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:58.804818 16235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:58.804944 16235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:58.805045 16235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:58.838534 16235 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:38665
I20260619 14:52:58.838594 16362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:38665 every 8 connection(s)
I20260619 14:52:58.840047 16235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:58.844712 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:58.842954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:43443)
0619 14:52:58.843684 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:58.843884 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":272,"threads_started":1}
W20260619 14:52:58.845432 16363 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:45075 (0 consecutive failures): Network error: Failed to ping master at 127.0.241.62:45075: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
I20260619 14:52:58.846172 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:58.845863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:42985)
0619 14:52:58.846008 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:58.846074 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 14:52:58.846992 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:58.846689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:46591)
0619 14:52:58.846820 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:58.846894 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 14:52:58.847219 16363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:52:58.849462 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 16235
I20260619 14:52:58.970817 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:58.970062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:33245)
0619 14:52:58.970501 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:58.970605 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":228,"threads_started":1}
I20260619 14:52:59.414160 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:59.413403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:36427)
0619 14:52:59.413851 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:59.413933 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":221,"threads_started":1}
I20260619 14:52:59.848589 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:59.847806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:54805)
0619 14:52:59.848246 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:59.848350 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":218,"threads_started":1}
I20260619 14:52:59.972381 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:59.971683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:56925)
0619 14:52:59.972139 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:59.972216 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":229,"threads_started":1}
I20260619 14:53:00.415803 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:00.415051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:43525)
0619 14:53:00.415543 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:00.415625 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":227,"threads_started":1}
I20260619 14:53:00.850440 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:00.849549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:49939)
0619 14:53:00.850191 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:00.850278 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":346,"threads_started":1}
I20260619 14:53:00.974145 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:00.973437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:41321)
0619 14:53:00.973878 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:00.973966 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":226,"threads_started":1}
I20260619 14:53:01.417465 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:01.416708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:39827)
0619 14:53:01.417207 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:01.417295 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":266,"threads_started":1}
I20260619 14:53:01.852263 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:01.851482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:58615)
0619 14:53:01.851998 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:01.852086 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":264,"threads_started":1}
I20260619 14:53:01.975870 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:01.975117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:36979)
0619 14:53:01.975578 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:01.975671 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":205,"threads_started":1}
I20260619 14:53:02.419492 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:02.418585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:40985)
0619 14:53:02.419068 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:02.419156 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":241,"threads_started":1}
I20260619 14:53:02.854195 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:02.853266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:54479)
0619 14:53:02.853852 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:02.853996 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":251,"threads_started":1}
I20260619 14:53:02.977593 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:02.976829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:59515)
0619 14:53:02.977325 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:02.977410 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":251,"threads_started":1}
I20260619 14:53:03.421378 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:03.420579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:34605)
0619 14:53:03.421122 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:03.421209 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":282,"threads_started":1}
I20260619 14:53:03.855926 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:03.855125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:34057)
0619 14:53:03.855588 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:03.855700 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":211,"threads_started":1}
I20260619 14:53:03.979032 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:03.978333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:42063)
0619 14:53:03.978775 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:03.978859 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":236,"threads_started":1}
I20260619 14:53:04.423002 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:04.422274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:36365)
0619 14:53:04.422722 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:04.422825 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":217,"threads_started":1}
I20260619 14:53:04.857451 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:04.856758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:54459)
0619 14:53:04.857202 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:04.857298 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":239,"threads_started":1}
I20260619 14:53:04.980861 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:04.979934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:51425)
0619 14:53:04.980557 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:04.980632 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":295,"threads_started":1}
I20260619 14:53:05.424803 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:05.423988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:44377)
0619 14:53:05.424494 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:05.424572 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":234,"threads_started":1}
I20260619 14:53:05.859185 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:05.858405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:59123)
0619 14:53:05.858876 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:05.858957 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":210,"threads_started":1}
I20260619 14:53:05.982388 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:05.981737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:49593)
0619 14:53:05.982135 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:05.982219 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":204,"threads_started":1}
I20260619 14:53:06.426671 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:06.425783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:54511)
0619 14:53:06.426424 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:06.426513 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":424,"threads_started":1}
I20260619 14:53:06.860877 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:06.860119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:51417)
0619 14:53:06.860572 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:06.860699 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260619 14:53:06.984036 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:06.983240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:45909)
0619 14:53:06.983784 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:06.983863 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":268,"threads_started":1}
I20260619 14:53:07.428408 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:07.427726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:55943)
0619 14:53:07.428144 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:07.428222 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":192,"threads_started":1}
I20260619 14:53:07.862375 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:07.861675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:40611)
0619 14:53:07.862116 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:07.862190 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":223,"threads_started":1}
I20260619 14:53:07.985677 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:07.984924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:55795)
0619 14:53:07.985396 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:07.985504 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":259,"threads_started":1}
I20260619 14:53:08.430086 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:08.429408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:43479)
0619 14:53:08.429850 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:08.429930 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260619 14:53:08.864058 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:08.863310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:44889)
0619 14:53:08.863821 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:08.863899 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":291,"threads_started":1}
I20260619 14:53:08.987418 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:08.986568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:51029)
0619 14:53:08.987097 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:08.987186 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":304,"threads_started":1}
I20260619 14:53:09.432160 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:09.431091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:56395)
0619 14:53:09.431786 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:09.431919 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":433,"threads_started":1}
I20260619 14:53:09.865559 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:09.864861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:34647)
0619 14:53:09.865284 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:09.865376 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":208,"threads_started":1}
I20260619 14:53:09.989060 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:09.988402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:43261)
0619 14:53:09.988831 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:09.988909 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":230,"threads_started":1}
I20260619 14:53:10.433851 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:10.433101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:33231)
0619 14:53:10.433570 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:10.433665 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":227,"threads_started":1}
I20260619 14:53:10.867165 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:10.866467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:48677)
0619 14:53:10.866916 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:10.867004 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":238,"threads_started":1}
I20260619 14:53:10.990803 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:10.989968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:58669)
0619 14:53:10.990508 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:10.990605 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":264,"threads_started":1}
I20260619 14:53:11.435883 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:11.434865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:54505)
0619 14:53:11.435357 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:11.435599 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":254,"threads_started":1}
I20260619 14:53:11.869015 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:11.868082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:43695)
0619 14:53:11.868612 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:11.868771 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":290,"threads_started":1}
I20260619 14:53:11.992455 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:11.991711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:57303)
0619 14:53:11.992186 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:11.992271 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":255,"threads_started":1}
I20260619 14:53:12.437561 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:12.436756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:39259)
0619 14:53:12.437274 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:12.437379 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":303,"threads_started":1}
I20260619 14:53:12.870752 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:12.870086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:56619)
0619 14:53:12.870508 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:12.870583 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":231,"threads_started":1}
I20260619 14:53:12.994422 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:12.993636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.1:57347)
0619 14:53:12.994110 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:12.994219 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
I20260619 14:53:13.439395 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:13.438655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.2:53355)
0619 14:53:13.439122 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:13.439218 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":255,"threads_started":1}
I20260619 14:53:13.856794 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 15939
I20260619 14:53:13.872707 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:13.871960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45075 (local address 127.0.241.3:44615)
0619 14:53:13.872443 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:13.872537 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:45075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":275,"threads_started":1}
I20260619 14:53:13.878712 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:13.858081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.0.241.1:45447)
0619 14:53:13.858772 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:13.858791 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:13.858845 (+ 54us) server_negotiation.cc:407] Connection header received
0619 14:53:13.860460 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:13.860470 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:13.860889 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:13.861183 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:13.862724 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:13.864060 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:13.865337 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:13.866012 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:13.866202 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:13.872433 (+ 6231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:13.872440 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:13.872457 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:13.872499 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:13.875342 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:13.876139 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:13.876146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:13.876152 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:13.876323 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:13.876705 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:13.876711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:13.876714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:13.877250 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:13.877607 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:13.878011 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 14:53:13.878437 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":298,"threads_started":1}
I20260619 14:53:13.880148 16096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:53:13.991377 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 15939
I20260619 14:53:14.008291 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 16104
I20260619 14:53:14.025406 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:14.009380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.0.241.2:40675)
0619 14:53:14.009931 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:14.009944 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:14.009981 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:53:14.011637 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:14.011646 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:14.012075 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:14.012349 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:14.013668 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:14.014872 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:14.016302 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:14.016940 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:14.017085 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:14.019647 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:14.019653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:14.019667 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:14.019702 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:14.022636 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:14.023203 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:14.023208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:14.023214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:14.023366 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:14.023717 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:14.023720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:14.023722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:14.024189 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:14.024427 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:14.024719 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 14:53:14.025129 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":264,"threads_started":1}
I20260619 14:53:14.026814 16227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:53:14.112808 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 16104
I20260619 14:53:14.131018 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 16235
I20260619 14:53:14.149201 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:14.132146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.0.241.3:38665)
0619 14:53:14.132766 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:14.132780 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:14.132828 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:53:14.134494 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:14.134503 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:14.134932 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:14.135303 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:14.136855 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:14.138081 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:14.139357 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:14.140057 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:14.140259 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:14.142856 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:14.142864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:14.142879 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:14.142915 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:14.145827 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:14.146526 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:14.146537 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:14.146550 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:14.146717 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:14.147095 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:14.147101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:14.147104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:14.147710 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:14.148054 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:14.148492 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 14:53:14.148907 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":294,"threads_started":1}
I20260619 14:53:14.150719 16358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:53:14.236697 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 16235
2026-06-19T14:53:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24326 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:53:14 dist-test-slave-p3ml krb5kdc[16420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:53:14 dist-test-slave-p3ml krb5kdc[16420](info): set up 2 sockets
Jun 19 14:53:14 dist-test-slave-p3ml krb5kdc[16420](info): commencing operation
krb5kdc: starting...
W20260619 14:53:15.344046 964 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:53:15 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:53:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:53:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:53:15.585170 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:37673
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:53:15.912869 16433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:15.913136 16433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:15.913228 16433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:15.920899 16433 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:53:15.921001 16433 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:53:15.921078 16433 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:53:15.921156 16433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:53:15.921236 16433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:15.921343 16433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:15.921412 16433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:53:15.921494 16433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:53:15.931959 16433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:37673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:53:15.933198 16433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:15.934499 16433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:15.946900 16438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:15.946946 16433 server_base.cc:1034] running on GCE node
W20260619 14:53:15.946946 16439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:15.946987 16441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:15.948186 16433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:15.949518 16433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:15.950763 16433 hybrid_clock.cc:648] HybridClock initialized: now 1781880795950710 us; error 35 us; skew 500 ppm
I20260619 14:53:15.951103 16433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:53:15 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:15.954347 16433 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:53:15.955865 16433 webserver.cc:460] Webserver started at http://127.0.241.62:40399/ using document root <none> and password file <none>
I20260619 14:53:15.956513 16433 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:15.956600 16433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:15.956840 16433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:53:15.959713 16433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "06789fbb3e9748fa8a144c8085bcf9fc"
format_stamp: "Formatted at 2026-06-19 14:53:15 on dist-test-slave-p3ml"
server_key: "746e7862ecc6743709246d351510fecd"
server_key_iv: "c79af1467fd59802820335394b50d306"
server_key_version: "encryptionkey@0"
I20260619 14:53:15.960422 16433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "06789fbb3e9748fa8a144c8085bcf9fc"
format_stamp: "Formatted at 2026-06-19 14:53:15 on dist-test-slave-p3ml"
server_key: "746e7862ecc6743709246d351510fecd"
server_key_iv: "c79af1467fd59802820335394b50d306"
server_key_version: "encryptionkey@0"
I20260619 14:53:15.965787 16433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 14:53:15.969161 16448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:15.970427 16433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:53:15.970610 16433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "06789fbb3e9748fa8a144c8085bcf9fc"
format_stamp: "Formatted at 2026-06-19 14:53:15 on dist-test-slave-p3ml"
server_key: "746e7862ecc6743709246d351510fecd"
server_key_iv: "c79af1467fd59802820335394b50d306"
server_key_version: "encryptionkey@0"
I20260619 14:53:15.970825 16433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:15.990037 16433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:15.994197 16433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:15.994488 16433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:16.013340 16433 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:37673
I20260619 14:53:16.013427 16499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:37673 every 8 connection(s)
I20260619 14:53:16.014675 16433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:53:16.017427 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 16433
I20260619 14:53:16.017642 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:53:16.018301 964 external_mini_cluster.cc:1428] Setting key 5e445248c6ec5e1d230e471f3f3ad4e7
I20260619 14:53:16.018748 16500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:53:16.034191 16500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Bootstrap starting.
Jun 19 14:53:16 dist-test-slave-p3ml krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:53:16.038609 16500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Neither blocks nor log segments found. Creating new log.
I20260619 14:53:16.040226 16500 log.cc:824] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:16.043911 16500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: No bootstrap required, opened a new log
I20260619 14:53:16.045133 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:16.020256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.0.241.62:37673)
0619 14:53:16.021005 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:16.021015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:16.021054 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:53:16.023404 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:16.023484 (+ 80us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:16.023859 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:16.024166 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:16.026135 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:16.027552 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:16.029624 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:16.030325 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:16.030481 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:16.035919 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:16.035932 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:16.035953 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:16.036016 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:16.040439 (+ 4423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:16.041332 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:16.041339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:16.041359 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:16.041553 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:16.041962 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:16.041967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:16.041971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:16.042645 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:16.042942 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:16.043308 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 14:53:16.043780 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":367,"threads_started":1}
I20260619 14:53:16.051854 16500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } }
I20260619 14:53:16.052199 16500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:53:16.052273 16500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06789fbb3e9748fa8a144c8085bcf9fc, State: Initialized, Role: FOLLOWER
I20260619 14:53:16.052743 16500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } }
I20260619 14:53:16.052943 16500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:53:16.053041 16500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:53:16.053161 16500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:53:16.055255 16500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } }
I20260619 14:53:16.055776 16500 leader_election.cc:302] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06789fbb3e9748fa8a144c8085bcf9fc; no voters:
I20260619 14:53:16.056305 16500 leader_election.cc:288] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:53:16.056514 16505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:53:16.057241 16505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 1 LEADER]: Becoming Leader. State: Replica: 06789fbb3e9748fa8a144c8085bcf9fc, State: Running, Role: LEADER
I20260619 14:53:16.057745 16505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } }
I20260619 14:53:16.058509 16500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:53:16.060320 16506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } } }
I20260619 14:53:16.060649 16506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:53:16.061529 16511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:53:16.061369 16507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06789fbb3e9748fa8a144c8085bcf9fc. Latest consensus state: current_term: 1 leader_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } } }
I20260619 14:53:16.062577 16507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:53:16.067350 16511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:53:16.076143 16511 catalog_manager.cc:1347] Generated new cluster ID: 20d029393d87406a9d6c0c59d030882f
I20260619 14:53:16.076272 16511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:53:16.093132 16511 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:53:16.094463 16511 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:53:16.101027 16511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Generated new TSK 0
I20260619 14:53:16.101822 16511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:53:16.167692 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:37673
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:53:16.502230 16526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:16.502499 16526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:16.502671 16526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:16.510460 16526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:16.510598 16526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:16.510784 16526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:53:16.520552 16526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:53:16.521838 16526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:16.523150 16526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:16.536105 16531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:16.536146 16534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:16.536190 16532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:16.537122 16526 server_base.cc:1034] running on GCE node
I20260619 14:53:16.537714 16526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:16.538681 16526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:16.539882 16526 hybrid_clock.cc:648] HybridClock initialized: now 1781880796539829 us; error 38 us; skew 500 ppm
I20260619 14:53:16.540210 16526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:53:16 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:16.543606 16526 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:53:16.545209 16526 webserver.cc:460] Webserver started at http://127.0.241.1:34271/ using document root <none> and password file <none>
I20260619 14:53:16.545914 16526 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:16.546065 16526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:16.546388 16526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:53:16.549330 16526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "2571b6cc039e4835898ee13d156e2225"
format_stamp: "Formatted at 2026-06-19 14:53:16 on dist-test-slave-p3ml"
server_key: "c8e6988fd8f967c1f209ce155e1dc595"
server_key_iv: "e087c6f6e12722a4f6665131c777e0f5"
server_key_version: "encryptionkey@0"
I20260619 14:53:16.550105 16526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "2571b6cc039e4835898ee13d156e2225"
format_stamp: "Formatted at 2026-06-19 14:53:16 on dist-test-slave-p3ml"
server_key: "c8e6988fd8f967c1f209ce155e1dc595"
server_key_iv: "e087c6f6e12722a4f6665131c777e0f5"
server_key_version: "encryptionkey@0"
I20260619 14:53:16.555181 16526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 14:53:16.558595 16541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:16.559986 16526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:53:16.560151 16526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "2571b6cc039e4835898ee13d156e2225"
format_stamp: "Formatted at 2026-06-19 14:53:16 on dist-test-slave-p3ml"
server_key: "c8e6988fd8f967c1f209ce155e1dc595"
server_key_iv: "e087c6f6e12722a4f6665131c777e0f5"
server_key_version: "encryptionkey@0"
I20260619 14:53:16.560366 16526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:16.590734 16526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:16.594868 16526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:16.595165 16526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:16.596243 16526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:16.597565 16526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:16.597669 16526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:16.597813 16526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:16.597893 16526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:16.631546 16526 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:45575
I20260619 14:53:16.631634 16653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:45575 every 8 connection(s)
I20260619 14:53:16.632879 16526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:53:16.633270 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 16526
I20260619 14:53:16.633476 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:53:16.634414 964 external_mini_cluster.cc:1428] Setting key e2ccb2a5f2d34debd823e43f7437efbf
Jun 19 14:53:16 dist-test-slave-p3ml krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:53:16.653371 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:16.635768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:49565 (local address 127.0.241.62:37673)
0619 14:53:16.636223 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:16.636229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:16.637416 (+ 1187us) server_negotiation.cc:407] Connection header received
0619 14:53:16.639244 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:16.639250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:16.639344 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:16.639511 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:16.640999 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:16.641711 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:16.643271 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:16.643748 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:16.643886 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:16.647536 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:16.647541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:16.647544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:16.647575 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:16.650139 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:16.650839 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:16.650846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:16.650849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:16.650973 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:16.651398 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:16.651403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:16.651406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:16.651724 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:16.651895 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:16.652642 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 14:53:16.652900 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":225,"threads_started":1}
I20260619 14:53:16.654050 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:16.636466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:49565)
0619 14:53:16.637265 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:16.637308 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:16.638981 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:16.639737 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:16.639745 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:16.640263 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:16.640827 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:16.640843 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:16.641878 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:16.641882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:16.643124 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:16.643135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:16.643985 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:16.643993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:16.644231 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:16.644913 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:16.644935 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:16.647286 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:16.650285 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:16.650293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:16.650309 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:16.650719 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:16.651084 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:16.651089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:16.651093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:16.651281 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:16.651862 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:16.651872 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:16.652215 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 14:53:16.652491 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 14:53:16.652946 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":375,"threads_started":1}
I20260619 14:53:16.655643 16654 heartbeater.cc:347] Connected to a master server at 127.0.241.62:37673
I20260619 14:53:16.655961 16654 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
I20260619 14:53:16.656675 16654 heartbeater.cc:511] Master 127.0.241.62:37673 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
I20260619 14:53:16.659032 16465 ts_manager.cc:194] Registered new tserver with Master: 2571b6cc039e4835898ee13d156e2225 (127.0.241.1:45575)
I20260619 14:53:16.661247 16465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:49565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:53:16.687295 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:37673
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:53:17.009053 16660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:17.009382 16660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:17.009546 16660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:17.016897 16660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:17.017055 16660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:17.017288 16660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:53:17.026845 16660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:53:17.028169 16660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:17.029413 16660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:17.042420 16665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:17.042382 16666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:17.042374 16668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:17.043303 16660 server_base.cc:1034] running on GCE node
I20260619 14:53:17.044001 16660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:17.044947 16660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:17.046175 16660 hybrid_clock.cc:648] HybridClock initialized: now 1781880797046125 us; error 45 us; skew 500 ppm
I20260619 14:53:17.046494 16660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:53:17 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:17.049698 16660 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:53:17.051141 16660 webserver.cc:460] Webserver started at http://127.0.241.2:42615/ using document root <none> and password file <none>
I20260619 14:53:17.051887 16660 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:17.052031 16660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:17.052350 16660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:53:17.055264 16660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "d0209daabca44ad592c24bbede31a28c"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-p3ml"
server_key: "abe4a32fa5d55789b7ebfbca799a605d"
server_key_iv: "34d762ba63d649700d113e2db184d291"
server_key_version: "encryptionkey@0"
I20260619 14:53:17.056100 16660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "d0209daabca44ad592c24bbede31a28c"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-p3ml"
server_key: "abe4a32fa5d55789b7ebfbca799a605d"
server_key_iv: "34d762ba63d649700d113e2db184d291"
server_key_version: "encryptionkey@0"
I20260619 14:53:17.061300 16660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:53:17.064945 16675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:17.066179 16660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:53:17.066347 16660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "d0209daabca44ad592c24bbede31a28c"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-p3ml"
server_key: "abe4a32fa5d55789b7ebfbca799a605d"
server_key_iv: "34d762ba63d649700d113e2db184d291"
server_key_version: "encryptionkey@0"
I20260619 14:53:17.066560 16660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:17.086091 16660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:17.090493 16660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:17.090793 16660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:17.091774 16660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:17.093443 16660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:17.093533 16660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:17.093675 16660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:17.093724 16660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:17.125326 16660 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:38959
I20260619 14:53:17.125373 16787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:38959 every 8 connection(s)
I20260619 14:53:17.126636 16660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:53:17.129438 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 16660
I20260619 14:53:17.129626 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:53:17.130311 964 external_mini_cluster.cc:1428] Setting key 81ce89058fff7da39dc1d1e053b04a77
Jun 19 14:53:17 dist-test-slave-p3ml krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:53:17.148555 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.129179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:40329 (local address 127.0.241.62:37673)
0619 14:53:17.129349 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:17.129353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:17.130821 (+ 1468us) server_negotiation.cc:407] Connection header received
0619 14:53:17.133241 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:17.133245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:17.133311 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:17.133400 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:17.135620 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.136753 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.138295 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.139062 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.139208 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.142691 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:17.142696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:17.142699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:17.142733 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:17.145408 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:17.146198 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:17.146210 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:17.146213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:17.146345 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:17.146775 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:17.146785 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:17.146787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:17.147045 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:17.147202 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:17.147967 (+ 765us) server_negotiation.cc:299] Negotiation successful
0619 14:53:17.148206 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:53:17.149524 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.129848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:40329)
0619 14:53:17.130639 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:17.130676 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:17.132907 (+ 2231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:17.133690 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:17.133704 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:17.134531 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:17.135375 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.135402 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.136898 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.136902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.138142 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.138153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.139277 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.139282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.139558 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.140248 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:17.140274 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:17.142456 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:17.145549 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:17.145558 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:17.145574 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:17.146073 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:17.146465 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:17.146470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:17.146473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:17.146653 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:17.147174 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:17.147182 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:17.147598 (+ 416us) client_negotiation.cc:769] Sending connection context
0619 14:53:17.147875 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 14:53:17.148405 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":306,"threads_started":1}
I20260619 14:53:17.150998 16788 heartbeater.cc:347] Connected to a master server at 127.0.241.62:37673
I20260619 14:53:17.151307 16788 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:17.152074 16788 heartbeater.cc:511] Master 127.0.241.62:37673 requested a full tablet report, sending...
I20260619 14:53:17.153528 16465 ts_manager.cc:194] Registered new tserver with Master: d0209daabca44ad592c24bbede31a28c (127.0.241.2:38959)
I20260619 14:53:17.154333 16465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:40329
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:53:17.186347 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:37673
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:53:17.524701 16793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:17.524964 16793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:17.525141 16793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:17.532444 16793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:17.532554 16793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:17.532780 16793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:53:17.542809 16793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:53:17.544430 16793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:17.545747 16793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:17.561264 16801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:17.561316 16799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:17.562211 16798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:17.562569 16793 server_base.cc:1034] running on GCE node
I20260619 14:53:17.563225 16793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:17.564471 16793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:17.565696 16793 hybrid_clock.cc:648] HybridClock initialized: now 1781880797565639 us; error 51 us; skew 500 ppm
I20260619 14:53:17.566041 16793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:53:17 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:17.570922 16793 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:53:17.576207 16793 webserver.cc:460] Webserver started at http://127.0.241.3:33073/ using document root <none> and password file <none>
I20260619 14:53:17.577250 16793 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:17.577425 16793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:17.577843 16793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:53:17.582316 16793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "ab90cb3957da44618c624bcdd8f063f4"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-p3ml"
server_key: "0f4bfe03c4aaf107313172944c344dca"
server_key_iv: "4ca4326ffff64e421eeef24a12aa8d00"
server_key_version: "encryptionkey@0"
I20260619 14:53:17.583317 16793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "ab90cb3957da44618c624bcdd8f063f4"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-p3ml"
server_key: "0f4bfe03c4aaf107313172944c344dca"
server_key_iv: "4ca4326ffff64e421eeef24a12aa8d00"
server_key_version: "encryptionkey@0"
I20260619 14:53:17.590363 16793 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260619 14:53:17.595486 16808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:17.597339 16793 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:53:17.597536 16793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "ab90cb3957da44618c624bcdd8f063f4"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-p3ml"
server_key: "0f4bfe03c4aaf107313172944c344dca"
server_key_iv: "4ca4326ffff64e421eeef24a12aa8d00"
server_key_version: "encryptionkey@0"
I20260619 14:53:17.597791 16793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:17.620430 16793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:17.625094 16793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:17.625416 16793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:17.626533 16793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:17.628028 16793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:17.628145 16793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:17.628317 16793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:17.628415 16793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:17.664485 16654 heartbeater.cc:503] Master 127.0.241.62:37673 was elected leader, sending a full tablet report...
I20260619 14:53:17.671368 16793 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:45875
I20260619 14:53:17.671481 16920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:45875 every 8 connection(s)
I20260619 14:53:17.672833 16793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:53:17.680485 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 16793
I20260619 14:53:17.680733 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:53:17.681353 964 external_mini_cluster.cc:1428] Setting key 2561d429ee80db2d1b1b58be661e67e0
Jun 19 14:53:17 dist-test-slave-p3ml krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:53:17.697008 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.675917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:33839 (local address 127.0.241.62:37673)
0619 14:53:17.676528 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:17.676537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:17.677575 (+ 1038us) server_negotiation.cc:407] Connection header received
0619 14:53:17.680796 (+ 3221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:17.680803 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:17.680921 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:17.681056 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:17.683457 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.684348 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.686120 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.686564 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.686719 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.690419 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:17.690426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:17.690433 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:17.690475 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:17.693362 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:17.694147 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:17.694155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:17.694159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:17.694308 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:17.694769 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:17.694777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:17.694781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:17.695071 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:17.695242 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:17.696137 (+ 895us) server_negotiation.cc:299] Negotiation successful
0619 14:53:17.696474 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":274,"threads_started":1}
I20260619 14:53:17.698129 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.676510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:33839)
0619 14:53:17.677384 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:17.677435 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:17.680418 (+ 2983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:17.681366 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:17.681380 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:17.682272 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:17.683143 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.683179 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.684516 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.684522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.685955 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.685969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.686831 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.686839 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.687091 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.687888 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:17.687913 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:17.690178 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:17.693518 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:17.693527 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:17.693544 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:17.694012 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:17.694427 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:17.694434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:17.694438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:17.694645 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:17.695223 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:17.695231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:17.695677 (+ 446us) client_negotiation.cc:769] Sending connection context
0619 14:53:17.696017 (+ 340us) client_negotiation.cc:240] Negotiation successful
0619 14:53:17.696572 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":322,"threads_started":1}
I20260619 14:53:17.699877 16921 heartbeater.cc:347] Connected to a master server at 127.0.241.62:37673
I20260619 14:53:17.700191 16921 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:17.700903 16921 heartbeater.cc:511] Master 127.0.241.62:37673 requested a full tablet report, sending...
I20260619 14:53:17.702438 16465 ts_manager.cc:194] Registered new tserver with Master: ab90cb3957da44618c624bcdd8f063f4 (127.0.241.3:45875)
I20260619 14:53:17.703246 16465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:33839
I20260619 14:53:17.709934 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:53:17.727180 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.714384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.0.241.62:37673)
0619 14:53:17.714575 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:17.714583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:17.714962 (+ 379us) server_negotiation.cc:407] Connection header received
0619 14:53:17.715190 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:17.715195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:17.715272 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:17.715376 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:17.716801 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.717536 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.718920 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.719320 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.719478 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.721839 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:17.721846 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:17.721850 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:17.721891 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:17.724522 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:17.725221 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:17.725228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:17.725231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:17.725361 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:17.725738 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:17.725746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:17.725750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:17.726065 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:17.726322 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:17.726631 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 14:53:17.726859 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:53:17.731110 16465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:53:17.737962 16465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:53:17.762182 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.750664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.0.241.1:45575)
0619 14:53:17.751843 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:17.751851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:17.752589 (+ 738us) server_negotiation.cc:407] Connection header received
0619 14:53:17.752891 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:17.752896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:17.753181 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:17.753353 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:17.754363 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.755969 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.759088 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.760624 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.760722 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.760941 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:17.761639 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 14:53:17.761817 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"thread_start_us":1194,"threads_started":1}
I20260619 14:53:17.763110 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.750339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45575 (local address 127.0.0.1:37060)
0619 14:53:17.752474 (+ 2135us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:17.752494 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:17.752697 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:17.753696 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:17.753703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:17.753731 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:17.754150 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.754165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.756140 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.756147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.758894 (+ 2747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.758910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.761156 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.761164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.761369 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.761432 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 14:53:17.762470 (+ 1038us) client_negotiation.cc:240] Negotiation successful
0619 14:53:17.762667 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1984,"mutex_wait_us":277,"thread_start_us":2322,"threads_started":1}
I20260619 14:53:17.764654 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.750160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:38959 (local address 127.0.0.1:42248)
0619 14:53:17.750979 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:17.751030 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:17.751183 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:17.754364 (+ 3181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:17.754368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:17.754392 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:17.754782 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.754794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.757962 (+ 3168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.757968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.761082 (+ 3114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.761093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.763769 (+ 2676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.763775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.763977 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.764010 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:53:17.764093 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 14:53:17.764254 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":883,"thread_start_us":696,"threads_started":1}
I20260619 14:53:17.765462 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.750425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.0.241.2:38959)
0619 14:53:17.752750 (+ 2325us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:17.752760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:17.752788 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:53:17.752887 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:17.752906 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:17.754135 (+ 1229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:17.756002 (+ 1867us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:17.756273 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.757793 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.761259 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.762708 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.764421 (+ 1713us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.764645 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:17.764718 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 14:53:17.765093 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2149,"thread_start_us":2026,"threads_started":1}
I20260619 14:53:17.767868 16589 tablet_service.cc:1467] Processing CreateTablet for tablet 2b68504e48d64d209a718d10b3b2c609 (DEFAULT_TABLE table=test-table [id=20672416877d4829830b0b3681769bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:53:17.768653 16723 tablet_service.cc:1467] Processing CreateTablet for tablet 2b68504e48d64d209a718d10b3b2c609 (DEFAULT_TABLE table=test-table [id=20672416877d4829830b0b3681769bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:53:17.769486 16589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b68504e48d64d209a718d10b3b2c609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:53:17.770071 16723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b68504e48d64d209a718d10b3b2c609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:53:17.772871 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.752793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:45875 (local address 127.0.0.1:59042)
0619 14:53:17.763330 (+ 10537us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:17.763350 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:17.763512 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:17.766089 (+ 2577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:17.766094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:17.766120 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:17.766271 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.766285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.768296 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.768301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.770144 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.770158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.772034 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.772041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.772233 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.772279 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 14:53:17.772378 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:53:17.772545 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10448,"thread_start_us":11315,"threads_started":1}
I20260619 14:53:17.773209 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.753464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.0.241.3:45875)
0619 14:53:17.765469 (+ 12005us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:17.765479 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:17.765514 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:53:17.765620 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:17.765628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:17.765873 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:17.766078 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:17.766482 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.768099 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.770314 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.771849 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.772000 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.772231 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:17.772465 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 14:53:17.772636 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11812,"thread_start_us":11640,"threads_started":1}
I20260619 14:53:17.777318 16856 tablet_service.cc:1467] Processing CreateTablet for tablet 2b68504e48d64d209a718d10b3b2c609 (DEFAULT_TABLE table=test-table [id=20672416877d4829830b0b3681769bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:53:17.778759 16856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b68504e48d64d209a718d10b3b2c609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:53:17.782979 16940 tablet_bootstrap.cc:492] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c: Bootstrap starting.
I20260619 14:53:17.784988 16941 tablet_bootstrap.cc:492] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225: Bootstrap starting.
I20260619 14:53:17.786899 16940 tablet_bootstrap.cc:654] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c: Neither blocks nor log segments found. Creating new log.
I20260619 14:53:17.788239 16940 log.cc:824] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:17.788473 16941 tablet_bootstrap.cc:654] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225: Neither blocks nor log segments found. Creating new log.
I20260619 14:53:17.789577 16941 log.cc:824] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:17.791857 16940 tablet_bootstrap.cc:492] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c: No bootstrap required, opened a new log
I20260619 14:53:17.792137 16940 ts_tablet_manager.cc:1397] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260619 14:53:17.792702 16944 tablet_bootstrap.cc:492] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4: Bootstrap starting.
I20260619 14:53:17.793519 16941 tablet_bootstrap.cc:492] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225: No bootstrap required, opened a new log
I20260619 14:53:17.793790 16941 ts_tablet_manager.cc:1397] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 14:53:17.795585 16944 tablet_bootstrap.cc:654] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4: Neither blocks nor log segments found. Creating new log.
I20260619 14:53:17.796504 16944 log.cc:824] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:17.798925 16944 tablet_bootstrap.cc:492] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4: No bootstrap required, opened a new log
I20260619 14:53:17.799185 16944 ts_tablet_manager.cc:1397] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260619 14:53:17.802613 16940 raft_consensus.cc:348] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } }
I20260619 14:53:17.803035 16940 raft_consensus.cc:374] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:53:17.803133 16940 raft_consensus.cc:729] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0209daabca44ad592c24bbede31a28c, State: Initialized, Role: FOLLOWER
I20260619 14:53:17.803750 16940 consensus_queue.cc:260] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } }
I20260619 14:53:17.804181 16941 raft_consensus.cc:348] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } }
I20260619 14:53:17.804701 16941 raft_consensus.cc:374] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:53:17.804798 16941 raft_consensus.cc:729] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2571b6cc039e4835898ee13d156e2225, State: Initialized, Role: FOLLOWER
I20260619 14:53:17.805016 16788 heartbeater.cc:503] Master 127.0.241.62:37673 was elected leader, sending a full tablet report...
I20260619 14:53:17.805591 16940 ts_tablet_manager.cc:1428] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260619 14:53:17.805429 16941 consensus_queue.cc:260] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } }
I20260619 14:53:17.807781 16944 raft_consensus.cc:348] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } }
I20260619 14:53:17.808190 16944 raft_consensus.cc:374] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:53:17.808287 16944 raft_consensus.cc:729] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90cb3957da44618c624bcdd8f063f4, State: Initialized, Role: FOLLOWER
I20260619 14:53:17.808320 16941 ts_tablet_manager.cc:1428] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260619 14:53:17.808844 16944 consensus_queue.cc:260] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } }
I20260619 14:53:17.809898 16921 heartbeater.cc:503] Master 127.0.241.62:37673 was elected leader, sending a full tablet report...
I20260619 14:53:17.810508 16944 ts_tablet_manager.cc:1428] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
I20260619 14:53:17.830577 16947 raft_consensus.cc:482] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:53:17.830840 16947 raft_consensus.cc:504] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } }
I20260619 14:53:17.832500 16947 leader_election.cc:288] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0209daabca44ad592c24bbede31a28c (127.0.241.2:38959), ab90cb3957da44618c624bcdd8f063f4 (127.0.241.3:45875)
I20260619 14:53:17.839802 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.832911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:35529 (local address 127.0.241.2:38959)
0619 14:53:17.833098 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:17.833104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:17.834134 (+ 1030us) server_negotiation.cc:407] Connection header received
0619 14:53:17.834317 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:17.834325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:17.834414 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:17.834484 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:17.835218 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.836025 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.837872 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.838749 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.838847 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.838918 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:17.839370 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 14:53:17.839526 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:53:17.839838 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.832981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:36013 (local address 127.0.241.3:45875)
0619 14:53:17.833183 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:17.833192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:17.834212 (+ 1020us) server_negotiation.cc:407] Connection header received
0619 14:53:17.834393 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:17.834396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:17.834451 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:17.834571 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:17.835299 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.836101 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.838090 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.838922 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.839052 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.839153 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:17.839414 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 14:53:17.839586 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:53:17.839962 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.832997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:38959 (local address 127.0.241.1:35529)
0619 14:53:17.834011 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:17.834027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:17.834143 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:17.834530 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:17.834537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:17.834565 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:17.835063 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.835077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.836161 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.836170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.837717 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.837730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.838977 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.838985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.839192 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.839226 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:53:17.839311 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:53:17.839498 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":1,"spinlock_wait_cycles":20864,"thread_start_us":509,"threads_started":1}
I20260619 14:53:17.840091 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.833002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:45875 (local address 127.0.241.1:36013)
0619 14:53:17.834061 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:17.834099 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:17.834261 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:17.834606 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:17.834612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:17.834643 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:17.835100 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.835110 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.836238 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.836241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.837935 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:17.837951 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.839124 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.839128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:17.839238 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.839260 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:53:17.839371 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:53:17.839525 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":51,"thread_start_us":628,"threads_started":1}
I20260619 14:53:17.840958 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b68504e48d64d209a718d10b3b2c609" candidate_uuid: "2571b6cc039e4835898ee13d156e2225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0209daabca44ad592c24bbede31a28c" is_pre_election: true
I20260619 14:53:17.841058 16876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b68504e48d64d209a718d10b3b2c609" candidate_uuid: "2571b6cc039e4835898ee13d156e2225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90cb3957da44618c624bcdd8f063f4" is_pre_election: true
I20260619 14:53:17.841481 16743 raft_consensus.cc:2454] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2571b6cc039e4835898ee13d156e2225 in term 0.
I20260619 14:53:17.841564 16876 raft_consensus.cc:2454] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2571b6cc039e4835898ee13d156e2225 in term 0.
I20260619 14:53:17.842216 16543 leader_election.cc:302] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2571b6cc039e4835898ee13d156e2225, ab90cb3957da44618c624bcdd8f063f4; no voters:
I20260619 14:53:17.842619 16947 raft_consensus.cc:2781] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:53:17.842767 16947 raft_consensus.cc:482] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:53:17.842888 16947 raft_consensus.cc:3037] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:53:17.845108 16947 raft_consensus.cc:504] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } }
I20260619 14:53:17.845765 16947 leader_election.cc:288] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [CANDIDATE]: Term 1 election: Requested vote from peers d0209daabca44ad592c24bbede31a28c (127.0.241.2:38959), ab90cb3957da44618c624bcdd8f063f4 (127.0.241.3:45875)
I20260619 14:53:17.846267 16743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b68504e48d64d209a718d10b3b2c609" candidate_uuid: "2571b6cc039e4835898ee13d156e2225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0209daabca44ad592c24bbede31a28c"
I20260619 14:53:17.846388 16876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b68504e48d64d209a718d10b3b2c609" candidate_uuid: "2571b6cc039e4835898ee13d156e2225" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90cb3957da44618c624bcdd8f063f4"
I20260619 14:53:17.846469 16743 raft_consensus.cc:3037] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:53:17.846619 16876 raft_consensus.cc:3037] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:53:17.848872 16743 raft_consensus.cc:2454] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2571b6cc039e4835898ee13d156e2225 in term 1.
I20260619 14:53:17.848919 16876 raft_consensus.cc:2454] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2571b6cc039e4835898ee13d156e2225 in term 1.
I20260619 14:53:17.849382 16543 leader_election.cc:302] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2571b6cc039e4835898ee13d156e2225, ab90cb3957da44618c624bcdd8f063f4; no voters:
I20260619 14:53:17.849675 16947 raft_consensus.cc:2781] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:53:17.850136 16947 raft_consensus.cc:686] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 1 LEADER]: Becoming Leader. State: Replica: 2571b6cc039e4835898ee13d156e2225, State: Running, Role: LEADER
I20260619 14:53:17.850636 16947 consensus_queue.cc:237] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } }
I20260619 14:53:17.855108 16464 catalog_manager.cc:5495] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2571b6cc039e4835898ee13d156e2225 (127.0.241.1). New cstate: current_term: 1 leader_uuid: "2571b6cc039e4835898ee13d156e2225" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } health_report { overall_health: UNKNOWN } } }
W20260619 14:53:17.878626 16789 tablet.cc:2369] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:53:17.885357 16655 tablet.cc:2369] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:53:17.885696 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.880079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.0.241.1:45575)
0619 14:53:17.880288 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:17.880293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:17.880312 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:53:17.880524 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:17.880529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:17.880605 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:17.880753 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:53:17.881136 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.881867 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.883338 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.883817 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.883954 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.884471 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:53:17.884959 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:53:17.885122 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:17.885231 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 14:53:17.885318 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:53:17.893499 16743 raft_consensus.cc:1261] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c [term 1 FOLLOWER]: Refusing update from remote peer 2571b6cc039e4835898ee13d156e2225: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:53:17.893505 16876 raft_consensus.cc:1261] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4 [term 1 FOLLOWER]: Refusing update from remote peer 2571b6cc039e4835898ee13d156e2225: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:53:17.894387 16947 consensus_queue.cc:1035] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:53:17.894665 16951 consensus_queue.cc:1035] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:53:17.907358 16955 mvcc.cc:205] Tried to move back new op lower bound from 7298583748162203648 to 7298583748000079872. Current Snapshot: MvccSnapshot[applied={T|T < 7298583748162203648}]
I20260619 14:53:17.922526 16464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:53:17.922801 16464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:53:17.926364 16464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Sending DeleteTablet for 3 replicas of tablet 2b68504e48d64d209a718d10b3b2c609
I20260619 14:53:17.927770 16856 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b68504e48d64d209a718d10b3b2c609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:53:17 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:59042
I20260619 14:53:17.927770 16723 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b68504e48d64d209a718d10b3b2c609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:53:17 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:42248
I20260619 14:53:17.928699 16589 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b68504e48d64d209a718d10b3b2c609 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:53:17 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:37060
I20260619 14:53:17.930478 16965 tablet_replica.cc:330] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c: stopping tablet replica
I20260619 14:53:17.931034 16964 tablet_replica.cc:330] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4: stopping tablet replica
I20260619 14:53:17.931195 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 16526
I20260619 14:53:17.931308 16966 tablet_replica.cc:330] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225: stopping tablet replica
I20260619 14:53:17.931730 16966 raft_consensus.cc:2229] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:53:17.932207 16965 raft_consensus.cc:2229] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:53:17.931792 16964 raft_consensus.cc:2229] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:53:17.932821 16964 raft_consensus.cc:2258] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:53:17.932981 16966 raft_consensus.cc:2258] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:53:17.933316 16965 raft_consensus.cc:2258] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:53:17.935493 16965 ts_tablet_manager.cc:1905] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:53:17.935509 16966 ts_tablet_manager.cc:1905] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:53:17.935503 16964 ts_tablet_manager.cc:1905] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:53:17.942464 16965 ts_tablet_manager.cc:1918] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:53:17.942631 16965 log.cc:1196] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/2b68504e48d64d209a718d10b3b2c609
I20260619 14:53:17.942687 16966 ts_tablet_manager.cc:1918] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:53:17.942541 16964 ts_tablet_manager.cc:1918] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:53:17.942881 16966 log.cc:1196] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/2b68504e48d64d209a718d10b3b2c609
I20260619 14:53:17.942929 16964 log.cc:1196] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/2b68504e48d64d209a718d10b3b2c609
I20260619 14:53:17.943143 16965 ts_tablet_manager.cc:1939] T 2b68504e48d64d209a718d10b3b2c609 P d0209daabca44ad592c24bbede31a28c: Deleting consensus metadata
I20260619 14:53:17.943454 16966 ts_tablet_manager.cc:1939] T 2b68504e48d64d209a718d10b3b2c609 P 2571b6cc039e4835898ee13d156e2225: Deleting consensus metadata
I20260619 14:53:17.943634 16964 ts_tablet_manager.cc:1939] T 2b68504e48d64d209a718d10b3b2c609 P ab90cb3957da44618c624bcdd8f063f4: Deleting consensus metadata
I20260619 14:53:17.945421 16452 catalog_manager.cc:4841] TS 2571b6cc039e4835898ee13d156e2225 (127.0.241.1:45575): tablet 2b68504e48d64d209a718d10b3b2c609 (table test-table [id=20672416877d4829830b0b3681769bb7]) successfully deleted
I20260619 14:53:17.945833 16450 catalog_manager.cc:4841] TS ab90cb3957da44618c624bcdd8f063f4 (127.0.241.3:45875): tablet 2b68504e48d64d209a718d10b3b2c609 (table test-table [id=20672416877d4829830b0b3681769bb7]) successfully deleted
I20260619 14:53:17.947547 16449 catalog_manager.cc:4841] TS d0209daabca44ad592c24bbede31a28c (127.0.241.2:38959): tablet 2b68504e48d64d209a718d10b3b2c609 (table test-table [id=20672416877d4829830b0b3681769bb7]) successfully deleted
Jun 19 14:53:17 dist-test-slave-p3ml krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:53:17.960343 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.932411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.0.241.1:45575)
0619 14:53:17.932614 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:17.932621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:17.934171 (+ 1550us) server_negotiation.cc:407] Connection header received
0619 14:53:17.943794 (+ 9623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:17.943801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:17.943900 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:17.943989 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:17.946148 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.947136 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.948828 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.949343 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.949548 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.954171 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:17.954182 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:17.954199 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:17.954235 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:17.957399 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:17.958245 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:17.958251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:17.958253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:17.958454 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:17.958843 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:17.958846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:17.958848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:17.959173 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:17.959383 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:17.959860 (+ 477us) server_negotiation.cc:299] Negotiation successful
0619 14:53:17.960030 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:53:17.961067 16649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:53:18.060587 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 16526
I20260619 14:53:18.079619 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 16660
Jun 19 14:53:18 dist-test-slave-p3ml krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:53:18.093739 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.080356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.0.241.2:38959)
0619 14:53:18.080536 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:18.080541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:18.080559 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:53:18.080695 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:18.080699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:18.080765 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:18.080851 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:18.082014 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.082739 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.084076 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.084447 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.084542 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.088626 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:18.088639 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:18.088655 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:18.088691 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:18.091325 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:18.091991 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:18.091999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:18.092001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:18.092128 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:18.092454 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:18.092457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:18.092460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:18.092811 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:18.093043 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:18.093289 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 14:53:18.093446 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:53:18.094385 16783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:53:18.183245 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 16660
I20260619 14:53:18.200029 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 16793
Jun 19 14:53:18 dist-test-slave-p3ml krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:53:18.215204 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.200806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59044 (local address 127.0.241.3:45875)
0619 14:53:18.200975 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:18.200979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:18.200999 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:53:18.201183 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:18.201187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:18.201254 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:18.201338 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:18.202577 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.203373 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.204800 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.205236 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.205327 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.209732 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:18.209746 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:18.209761 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:18.209795 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:18.212729 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:18.213337 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:18.213343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:18.213346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:18.213490 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:18.213858 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:18.213862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:18.213864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:18.214279 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:18.214543 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:18.214746 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 14:53:18.214931 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:53:18.215867 16916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:53:18.304373 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 16793
I20260619 14:53:18.320971 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 16433
I20260619 14:53:18.322073 16495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:53:18.431715 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 16433
I20260619 14:53:38.453054 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:37673
--webserver_interface=127.0.241.62
--webserver_port=40399
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:53:38.782052 16972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:38.782320 16972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:38.782490 16972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:38.793475 16972 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:53:38.793567 16972 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:53:38.793634 16972 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:53:38.793704 16972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:53:38.793780 16972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:38.793910 16972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:38.793974 16972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:53:38.794051 16972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:53:38.804112 16972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:37673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=40399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:53:38.805375 16972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:38.806762 16972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:38.829371 16978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:38.829406 16981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:38.829329 16979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:38.830124 16972 server_base.cc:1034] running on GCE node
I20260619 14:53:38.831019 16972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:38.832588 16972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:38.833839 16972 hybrid_clock.cc:648] HybridClock initialized: now 1781880818833787 us; error 30 us; skew 500 ppm
I20260619 14:53:38.834240 16972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:53:38 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:38.837841 16972 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:53:38.839646 16972 webserver.cc:460] Webserver started at http://127.0.241.62:40399/ using document root <none> and password file <none>
I20260619 14:53:38.840469 16972 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:38.840622 16972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:38.846669 16972 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:53:38.849682 16988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:38.850929 16972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:53:38.851109 16972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "06789fbb3e9748fa8a144c8085bcf9fc"
format_stamp: "Formatted at 2026-06-19 14:53:15 on dist-test-slave-p3ml"
server_key: "746e7862ecc6743709246d351510fecd"
server_key_iv: "c79af1467fd59802820335394b50d306"
server_key_version: "encryptionkey@0"
I20260619 14:53:38.852316 16972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:38.868633 16972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:38.872712 16972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:38.873015 16972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:38.892481 16972 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:37673
I20260619 14:53:38.892509 17039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:37673 every 8 connection(s)
I20260619 14:53:38.894111 16972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:53:38.895973 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 16972
I20260619 14:53:38.897010 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:45575
--local_ip_for_outbound_sockets=127.0.241.1
--tserver_master_addrs=127.0.241.62:37673
--webserver_port=34271
--webserver_interface=127.0.241.1
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:53:38.904315 17040 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:53:38.908381 17040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Bootstrap starting.
I20260619 14:53:38.928512 17040 log.cc:824] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:38.940469 17040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:53:38.941038 17040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Bootstrap complete.
I20260619 14:53:38.950299 17040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } }
I20260619 14:53:38.951105 17040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06789fbb3e9748fa8a144c8085bcf9fc, State: Initialized, Role: FOLLOWER
I20260619 14:53:38.951799 17040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } }
I20260619 14:53:38.952049 17040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:53:38.952168 17040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:53:38.952361 17040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:53:38.956305 17040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } }
I20260619 14:53:38.956888 17040 leader_election.cc:302] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06789fbb3e9748fa8a144c8085bcf9fc; no voters:
I20260619 14:53:38.957615 17040 leader_election.cc:288] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:53:38.957749 17044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:53:38.959837 17044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 2 LEADER]: Becoming Leader. State: Replica: 06789fbb3e9748fa8a144c8085bcf9fc, State: Running, Role: LEADER
I20260619 14:53:38.959944 17040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:53:38.960405 17044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } }
I20260619 14:53:38.962852 17045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } } }
I20260619 14:53:38.963197 17045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:53:38.963677 17046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06789fbb3e9748fa8a144c8085bcf9fc. Latest consensus state: current_term: 2 leader_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } } }
I20260619 14:53:38.963964 17046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:53:38.965911 17053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:53:38.975353 17053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:53:38.977144 17053 catalog_manager.cc:1259] Loaded cluster ID: 20d029393d87406a9d6c0c59d030882f
I20260619 14:53:38.977273 17053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:53:38.982313 17053 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:53:38.984270 17053 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Loaded TSK: 0
I20260619 14:53:38.998080 17061 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:53:39.012951 17053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Generated new TSK 1
I20260619 14:53:39.013309 17053 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:53:39.277108 17042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:39.277371 17042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:39.277524 17042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:39.284889 17042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:39.285006 17042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:39.285234 17042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:53:39.295089 17042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:45575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=34271
--tserver_master_addrs=127.0.241.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:53:39.296408 17042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:39.297749 17042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:39.310892 17068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:39.311213 17070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:39.311841 17067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:39.312464 17042 server_base.cc:1034] running on GCE node
I20260619 14:53:39.313150 17042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:39.314159 17042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:39.315402 17042 hybrid_clock.cc:648] HybridClock initialized: now 1781880819315342 us; error 52 us; skew 500 ppm
I20260619 14:53:39.315790 17042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:53:39 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:39.322624 17042 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:53:39.324257 17042 webserver.cc:460] Webserver started at http://127.0.241.1:34271/ using document root <none> and password file <none>
I20260619 14:53:39.324958 17042 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:39.325116 17042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:39.330536 17042 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:53:39.333245 17077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:39.334656 17042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:53:39.334812 17042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "2571b6cc039e4835898ee13d156e2225"
format_stamp: "Formatted at 2026-06-19 14:53:16 on dist-test-slave-p3ml"
server_key: "c8e6988fd8f967c1f209ce155e1dc595"
server_key_iv: "e087c6f6e12722a4f6665131c777e0f5"
server_key_version: "encryptionkey@0"
I20260619 14:53:39.335805 17042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:39.379155 17042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:39.383731 17042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:39.384011 17042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:39.384976 17042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:39.386286 17042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:39.386370 17042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:39.386502 17042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:39.386600 17042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:39.420186 17042 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:45575
I20260619 14:53:39.420279 17189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:45575 every 8 connection(s)
I20260619 14:53:39.421852 17042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:53:39.423988 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 17042
I20260619 14:53:39.424752 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:38959
--local_ip_for_outbound_sockets=127.0.241.2
--tserver_master_addrs=127.0.241.62:37673
--webserver_port=42615
--webserver_interface=127.0.241.2
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:53:39 dist-test-slave-p3ml krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:53:39.447371 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:39.424730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:60851 (local address 127.0.241.62:37673)
0619 14:53:39.425561 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:39.425573 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:39.426188 (+ 615us) server_negotiation.cc:407] Connection header received
0619 14:53:39.428375 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:39.428386 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:39.428709 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:39.429057 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:39.430780 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:39.432148 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:39.434710 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:39.435350 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:39.435549 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:39.439802 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:39.439812 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:39.439826 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:39.439862 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:39.442854 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:39.443698 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:39.443705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:39.443710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:39.443875 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:39.444320 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:39.444332 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:39.444334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:39.444797 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:39.445010 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:39.445771 (+ 761us) server_negotiation.cc:299] Negotiation successful
0619 14:53:39.446171 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":514,"threads_started":1}
I20260619 14:53:39.447371 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:39.424992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:60851)
0619 14:53:39.426021 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:39.426057 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:39.427852 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:39.429211 (+ 1359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:39.429223 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:39.429828 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:39.430574 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:39.430593 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:39.432315 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:39.432323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:39.434558 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:39.434582 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:39.435608 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:39.435618 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:39.435943 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:39.436648 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:39.436681 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:39.439504 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:39.443017 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:39.443026 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:39.443041 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:39.443551 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:39.443997 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:39.444002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:39.444006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:39.444191 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:39.444936 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:39.444944 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:39.445281 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 14:53:39.445538 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 14:53:39.446068 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":625,"threads_started":1}
I20260619 14:53:39.450596 17190 heartbeater.cc:347] Connected to a master server at 127.0.241.62:37673
I20260619 14:53:39.450903 17190 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:39.451669 17190 heartbeater.cc:511] Master 127.0.241.62:37673 requested a full tablet report, sending...
I20260619 14:53:39.453929 17005 ts_manager.cc:194] Registered new tserver with Master: 2571b6cc039e4835898ee13d156e2225 (127.0.241.1:45575)
I20260619 14:53:39.456028 17005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:60851
W20260619 14:53:39.753756 17194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:39.754065 17194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:39.754168 17194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:39.761547 17194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:39.761643 17194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:39.761754 17194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:53:39.771329 17194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:38959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=42615
--tserver_master_addrs=127.0.241.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:53:39.772500 17194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:39.773671 17194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:39.786358 17202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:39.786422 17200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:39.786419 17199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:39.787297 17194 server_base.cc:1034] running on GCE node
I20260619 14:53:39.787945 17194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:39.788859 17194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:39.790064 17194 hybrid_clock.cc:648] HybridClock initialized: now 1781880819790030 us; error 29 us; skew 500 ppm
I20260619 14:53:39.790361 17194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:53:39 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:39.793321 17194 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:53:39.794634 17194 webserver.cc:460] Webserver started at http://127.0.241.2:42615/ using document root <none> and password file <none>
I20260619 14:53:39.795261 17194 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:39.795413 17194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:39.800653 17194 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:53:39.803169 17209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:39.804409 17194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:53:39.804577 17194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "d0209daabca44ad592c24bbede31a28c"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-p3ml"
server_key: "abe4a32fa5d55789b7ebfbca799a605d"
server_key_iv: "34d762ba63d649700d113e2db184d291"
server_key_version: "encryptionkey@0"
I20260619 14:53:39.805510 17194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:39.824925 17194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:39.828709 17194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:39.828974 17194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:39.829914 17194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:39.831120 17194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:39.831252 17194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:39.831382 17194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:39.831494 17194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:39.865010 17194 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:38959
I20260619 14:53:39.865104 17321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:38959 every 8 connection(s)
I20260619 14:53:39.866406 17194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:53:39.869628 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 17194
I20260619 14:53:39.870462 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:45875
--local_ip_for_outbound_sockets=127.0.241.3
--tserver_master_addrs=127.0.241.62:37673
--webserver_port=33073
--webserver_interface=127.0.241.3
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:53:39 dist-test-slave-p3ml krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:53:39.888633 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:39.869025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:51151 (local address 127.0.241.62:37673)
0619 14:53:39.869251 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:39.869256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:39.870536 (+ 1280us) server_negotiation.cc:407] Connection header received
0619 14:53:39.872506 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:39.872511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:39.872600 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:39.872724 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:39.874188 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:39.875043 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:39.877541 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:39.877951 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:39.878049 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:39.881585 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:39.881594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:39.881601 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:39.881644 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:39.885235 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:39.885950 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:39.885958 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:39.885965 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:39.886114 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:39.886572 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:39.886580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:39.886586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:39.886990 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:39.887147 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:39.887889 (+ 742us) server_negotiation.cc:299] Negotiation successful
0619 14:53:39.888158 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:53:39.889237 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:39.869571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:51151)
0619 14:53:39.870365 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:39.870402 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:39.872238 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:39.872958 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:39.872966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:39.873451 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:39.874003 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:39.874020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:39.875225 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:39.875233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:39.877372 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:39.877390 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:39.878203 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:39.878211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:39.878462 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:39.879161 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:39.879190 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:39.881336 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:39.885379 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:39.885385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:39.885396 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:39.885803 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:39.886246 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:39.886250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:39.886253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:39.886449 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:39.887122 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:39.887128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:39.887498 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 14:53:39.887740 (+ 242us) client_negotiation.cc:240] Negotiation successful
0619 14:53:39.888109 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":330,"threads_started":1}
I20260619 14:53:39.890621 17322 heartbeater.cc:347] Connected to a master server at 127.0.241.62:37673
I20260619 14:53:39.890928 17322 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:39.891716 17322 heartbeater.cc:511] Master 127.0.241.62:37673 requested a full tablet report, sending...
I20260619 14:53:39.893183 17005 ts_manager.cc:194] Registered new tserver with Master: d0209daabca44ad592c24bbede31a28c (127.0.241.2:38959)
I20260619 14:53:39.894032 17005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:51151
W20260619 14:53:40.206552 17325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:40.206809 17325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:40.206982 17325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:40.214102 17325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:40.214222 17325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:40.214398 17325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:53:40.224292 17325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:45875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=33073
--tserver_master_addrs=127.0.241.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:53:40.225780 17325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:40.227270 17325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:40.240898 17330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:40.240952 17331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:40.240978 17333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:40.241839 17325 server_base.cc:1034] running on GCE node
I20260619 14:53:40.242372 17325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:40.243196 17325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:40.244424 17325 hybrid_clock.cc:648] HybridClock initialized: now 1781880820244333 us; error 66 us; skew 500 ppm
I20260619 14:53:40.244721 17325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:53:40 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:40.247695 17325 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:53:40.249058 17325 webserver.cc:460] Webserver started at http://127.0.241.3:33073/ using document root <none> and password file <none>
I20260619 14:53:40.249689 17325 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:40.249800 17325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:40.254912 17325 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:53:40.257637 17340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:40.258915 17325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:53:40.259083 17325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "ab90cb3957da44618c624bcdd8f063f4"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-p3ml"
server_key: "0f4bfe03c4aaf107313172944c344dca"
server_key_iv: "4ca4326ffff64e421eeef24a12aa8d00"
server_key_version: "encryptionkey@0"
I20260619 14:53:40.260150 17325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:40.290005 17325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:40.294802 17325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:40.295096 17325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:40.296172 17325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:40.297462 17325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:40.297560 17325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:40.297631 17325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:40.297695 17325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:40.331286 17325 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:45875
I20260619 14:53:40.331386 17452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:45875 every 8 connection(s)
I20260619 14:53:40.332736 17325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:53:40.336382 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 17325
Jun 19 14:53:40 dist-test-slave-p3ml krb5kdc[16420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:53:40.351872 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.337293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.0.241.62:37673)
0619 14:53:40.338073 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.338079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.338096 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:53:40.338231 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.338235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.338298 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.338420 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:40.340180 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.340920 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.342337 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.342777 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.342917 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.346185 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:40.346189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:40.346192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:40.346220 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:40.349141 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:40.349797 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:40.349803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:40.349808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:40.349939 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:40.350303 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:40.350309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:40.350313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:40.350581 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:40.350863 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:40.351185 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 14:53:40.351419 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":568,"threads_started":1}
I20260619 14:53:40.353971 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.335375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:50625 (local address 127.0.241.62:37673)
0619 14:53:40.335591 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.335597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.336928 (+ 1331us) server_negotiation.cc:407] Connection header received
0619 14:53:40.339165 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.339179 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.339264 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.339377 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:40.340953 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.341685 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.343197 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.343659 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.343787 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.347504 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:40.347509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:40.347513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:40.347555 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:40.350555 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:40.351305 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:40.351312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:40.351315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:40.351508 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:40.351970 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:40.351978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:40.351982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:40.352312 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:40.352517 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:40.353319 (+ 802us) server_negotiation.cc:299] Negotiation successful
0619 14:53:40.353539 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:53:40.355149 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.335891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:50625)
0619 14:53:40.336748 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:40.336800 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:40.338907 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:40.339643 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:40.339651 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:40.340141 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:40.340788 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:40.340806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.341841 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.341846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:40.343036 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:40.343048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.343883 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.343890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:40.344114 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.344792 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:40.344813 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:40.347249 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:40.350707 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:40.350716 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:40.350742 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:40.351174 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:40.351641 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:40.351646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:40.351650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:40.351847 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:40.352491 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:40.352500 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:40.352924 (+ 424us) client_negotiation.cc:769] Sending connection context
0619 14:53:40.353244 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 14:53:40.353777 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":357,"threads_started":1}
I20260619 14:53:40.356508 17453 heartbeater.cc:347] Connected to a master server at 127.0.241.62:37673
I20260619 14:53:40.356784 17453 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:40.357471 17453 heartbeater.cc:511] Master 127.0.241.62:37673 requested a full tablet report, sending...
I20260619 14:53:40.358867 17005 ts_manager.cc:194] Registered new tserver with Master: ab90cb3957da44618c624bcdd8f063f4 (127.0.241.3:45875)
I20260619 14:53:40.359776 17005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:50625
I20260619 14:53:40.366779 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:53:40.373152 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.367748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.0.241.62:37673)
0619 14:53:40.367945 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.367955 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.368401 (+ 446us) server_negotiation.cc:407] Connection header received
0619 14:53:40.368537 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.368541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.368678 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.368800 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:53:40.369155 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.369896 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.371373 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.371887 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.371989 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.372454 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:53:40.372639 (+ 185us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 14:53:40.372927 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36848: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 14:53:40.373276 17192 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36848: token expired
W20260619 14:53:40.373796 16928 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.241.62:37673)
I20260619 14:53:40.386157 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.374220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.0.241.62:37673)
0619 14:53:40.374402 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.374406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.374443 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:53:40.374580 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.374583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.374646 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.374737 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:40.376201 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.376879 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.378303 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.378687 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.378774 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.381121 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:40.381126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:40.381129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:40.381157 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:40.383815 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:40.384477 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:40.384481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:40.384484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:40.384608 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:40.384938 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:40.384942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:40.384944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:40.385242 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:40.385417 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:40.385668 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 14:53:40.385816 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:53:40.389309 17005 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:53:40.396543 17005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:53:40.435093 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.409508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.0.241.2:38959)
0619 14:53:40.425112 (+ 15604us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.425124 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.426401 (+ 1277us) server_negotiation.cc:407] Connection header received
0619 14:53:40.426543 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.426554 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.426879 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.427104 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:40.428226 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.429910 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.432738 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.433823 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.433985 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.434187 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:40.434606 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 14:53:40.434771 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15454,"thread_start_us":15325,"threads_started":1}
I20260619 14:53:40.435140 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.409074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:38959 (local address 127.0.0.1:55600)
0619 14:53:40.426078 (+ 17004us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:40.426106 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:40.426244 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:40.427424 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:40.427433 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:40.427497 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:40.427971 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:40.427987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.430087 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.430091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:40.432535 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:40.432552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.434057 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.434063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:40.434375 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.434413 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:53:40.434548 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 14:53:40.434728 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16826,"mutex_wait_us":60,"thread_start_us":16668,"threads_started":1}
I20260619 14:53:40.437815 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.408293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45575 (local address 127.0.0.1:43150)
0619 14:53:40.423694 (+ 15401us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:40.423716 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:40.424564 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:40.429609 (+ 5045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:40.429614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:40.429638 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:40.429977 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:40.429993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.431944 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.431950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:40.434429 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:40.434444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.437168 (+ 2724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.437173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:40.437297 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.437322 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:53:40.437417 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 14:53:40.437508 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15249,"mutex_wait_us":92,"thread_start_us":17649,"threads_started":1}
I20260619 14:53:40.437958 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.408999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43150 (local address 127.0.241.1:45575)
0619 14:53:40.429065 (+ 20066us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.429075 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.429105 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:53:40.429196 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.429202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.429403 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.429595 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:40.430165 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.431743 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.435618 (+ 3875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.436933 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.437066 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.437290 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:40.437483 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 14:53:40.437662 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19901,"thread_start_us":19773,"threads_started":1}
I20260619 14:53:40.439531 17257 tablet_service.cc:1467] Processing CreateTablet for tablet 97969f26a258425b894518c24cfe420b (DEFAULT_TABLE table=test-table [id=7e4e9ae782f84082a4de20eff9d91823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:53:40.440317 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.408143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:45875 (local address 127.0.0.1:39960)
0619 14:53:40.423051 (+ 14908us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:40.423097 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:40.423269 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:40.429967 (+ 6698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:40.429972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:40.429998 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:40.430425 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:40.430435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.431802 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.431808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:40.434429 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:40.434444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.436423 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.436431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:40.439670 (+ 3239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.439705 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:53:40.439792 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:53:40.439921 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14670,"mutex_wait_us":177,"thread_start_us":15762,"threads_started":1}
I20260619 14:53:40.441116 17257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97969f26a258425b894518c24cfe420b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:53:40.441267 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.408378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.0.241.3:45875)
0619 14:53:40.429426 (+ 21048us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.429436 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.429469 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:53:40.429559 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.429566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.429770 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.429970 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:40.430598 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.431677 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.434630 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.436189 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.436328 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.436544 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:40.440832 (+ 4288us) server_negotiation.cc:299] Negotiation successful
0619 14:53:40.440964 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20883,"thread_start_us":20709,"threads_started":1}
I20260619 14:53:40.441533 17125 tablet_service.cc:1467] Processing CreateTablet for tablet 97969f26a258425b894518c24cfe420b (DEFAULT_TABLE table=test-table [id=7e4e9ae782f84082a4de20eff9d91823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:53:40.442941 17125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97969f26a258425b894518c24cfe420b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:53:40.444125 17388 tablet_service.cc:1467] Processing CreateTablet for tablet 97969f26a258425b894518c24cfe420b (DEFAULT_TABLE table=test-table [id=7e4e9ae782f84082a4de20eff9d91823]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:53:40.445206 17388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97969f26a258425b894518c24cfe420b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:53:40.456211 17469 tablet_bootstrap.cc:492] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4: Bootstrap starting.
I20260619 14:53:40.459672 17190 heartbeater.cc:503] Master 127.0.241.62:37673 was elected leader, sending a full tablet report...
I20260619 14:53:40.460522 17469 tablet_bootstrap.cc:654] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4: Neither blocks nor log segments found. Creating new log.
I20260619 14:53:40.462821 17470 tablet_bootstrap.cc:492] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c: Bootstrap starting.
I20260619 14:53:40.463393 17469 log.cc:824] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:40.464882 17471 tablet_bootstrap.cc:492] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225: Bootstrap starting.
I20260619 14:53:40.466504 17469 tablet_bootstrap.cc:492] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4: No bootstrap required, opened a new log
I20260619 14:53:40.466768 17469 ts_tablet_manager.cc:1397] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260619 14:53:40.467412 17470 tablet_bootstrap.cc:654] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c: Neither blocks nor log segments found. Creating new log.
I20260619 14:53:40.468452 17470 log.cc:824] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:40.470850 17471 tablet_bootstrap.cc:654] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225: Neither blocks nor log segments found. Creating new log.
I20260619 14:53:40.470918 17470 tablet_bootstrap.cc:492] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c: No bootstrap required, opened a new log
I20260619 14:53:40.471240 17470 ts_tablet_manager.cc:1397] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260619 14:53:40.472095 17471 log.cc:824] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:40.477603 17471 tablet_bootstrap.cc:492] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225: No bootstrap required, opened a new log
I20260619 14:53:40.477878 17471 ts_tablet_manager.cc:1397] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260619 14:53:40.477756 17469 raft_consensus.cc:348] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } }
I20260619 14:53:40.478232 17469 raft_consensus.cc:374] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:53:40.478336 17469 raft_consensus.cc:729] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab90cb3957da44618c624bcdd8f063f4, State: Initialized, Role: FOLLOWER
I20260619 14:53:40.478968 17469 consensus_queue.cc:260] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } }
I20260619 14:53:40.479303 17470 raft_consensus.cc:348] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } }
I20260619 14:53:40.479838 17470 raft_consensus.cc:374] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:53:40.480103 17453 heartbeater.cc:503] Master 127.0.241.62:37673 was elected leader, sending a full tablet report...
I20260619 14:53:40.480751 17469 ts_tablet_manager.cc:1428] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260619 14:53:40.481079 17470 raft_consensus.cc:729] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0209daabca44ad592c24bbede31a28c, State: Initialized, Role: FOLLOWER
I20260619 14:53:40.481675 17470 consensus_queue.cc:260] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } }
I20260619 14:53:40.482759 17322 heartbeater.cc:503] Master 127.0.241.62:37673 was elected leader, sending a full tablet report...
I20260619 14:53:40.483345 17470 ts_tablet_manager.cc:1428] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260619 14:53:40.488231 17471 raft_consensus.cc:348] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } }
I20260619 14:53:40.488683 17471 raft_consensus.cc:374] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:53:40.488794 17471 raft_consensus.cc:729] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2571b6cc039e4835898ee13d156e2225, State: Initialized, Role: FOLLOWER
I20260619 14:53:40.489284 17471 consensus_queue.cc:260] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } }
I20260619 14:53:40.490782 17471 ts_tablet_manager.cc:1428] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 14:53:40.579241 17476 raft_consensus.cc:482] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:53:40.579638 17476 raft_consensus.cc:504] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } }
I20260619 14:53:40.581543 17476 leader_election.cc:288] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab90cb3957da44618c624bcdd8f063f4 (127.0.241.3:45875), 2571b6cc039e4835898ee13d156e2225 (127.0.241.1:45575)
W20260619 14:53:40.584395 17454 tablet.cc:2369] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:53:40.591156 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.582016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:48059 (local address 127.0.241.3:45875)
0619 14:53:40.582254 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.582264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.583560 (+ 1296us) server_negotiation.cc:407] Connection header received
0619 14:53:40.583775 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.583779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.583836 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.583944 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:40.584762 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.585564 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.588617 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.589938 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.590105 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.590222 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:40.590754 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 14:53:40.590889 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 14:53:40.591205 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.582089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:49019 (local address 127.0.241.1:45575)
0619 14:53:40.582318 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.582327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.583476 (+ 1149us) server_negotiation.cc:407] Connection header received
0619 14:53:40.583716 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.583723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.583811 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.583889 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:40.584650 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.585658 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.588625 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.590004 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.590157 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.590259 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:40.590754 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 14:53:40.590889 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 14:53:40.591487 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.582089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:45875 (local address 127.0.241.2:48059)
0619 14:53:40.583422 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:40.583443 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:40.583630 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:40.583989 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:40.583995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:40.584026 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:40.584526 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:40.584541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.585713 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.585721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:40.588440 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:40.588455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.590196 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.590205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:40.590493 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.590615 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 14:53:40.590722 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:53:40.590917 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187,"mutex_wait_us":283,"thread_start_us":831,"threads_started":1}
I20260619 14:53:40.591528 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.581862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45575 (local address 127.0.241.2:49019)
0619 14:53:40.583322 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:40.583338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:40.583538 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:40.583932 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:40.583936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:40.583960 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:40.584445 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:40.584461 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.585804 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.585810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:40.588440 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:40.588455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.590223 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.590229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:40.590495 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.590611 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 14:53:40.590711 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:53:40.590917 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1336,"mutex_wait_us":292,"spinlock_wait_cycles":31104,"thread_start_us":1175,"threads_started":1}
I20260619 14:53:40.592558 17408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97969f26a258425b894518c24cfe420b" candidate_uuid: "d0209daabca44ad592c24bbede31a28c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90cb3957da44618c624bcdd8f063f4" is_pre_election: true
I20260619 14:53:40.592599 17145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97969f26a258425b894518c24cfe420b" candidate_uuid: "d0209daabca44ad592c24bbede31a28c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2571b6cc039e4835898ee13d156e2225" is_pre_election: true
I20260619 14:53:40.593050 17408 raft_consensus.cc:2454] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0209daabca44ad592c24bbede31a28c in term 0.
I20260619 14:53:40.593168 17145 raft_consensus.cc:2454] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0209daabca44ad592c24bbede31a28c in term 0.
I20260619 14:53:40.593753 17211 leader_election.cc:302] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab90cb3957da44618c624bcdd8f063f4, d0209daabca44ad592c24bbede31a28c; no voters:
I20260619 14:53:40.594174 17476 raft_consensus.cc:2781] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:53:40.594333 17476 raft_consensus.cc:482] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:53:40.594439 17476 raft_consensus.cc:3037] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:53:40.596657 17476 raft_consensus.cc:504] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } }
I20260619 14:53:40.597236 17476 leader_election.cc:288] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [CANDIDATE]: Term 1 election: Requested vote from peers ab90cb3957da44618c624bcdd8f063f4 (127.0.241.3:45875), 2571b6cc039e4835898ee13d156e2225 (127.0.241.1:45575)
I20260619 14:53:40.597615 17408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97969f26a258425b894518c24cfe420b" candidate_uuid: "d0209daabca44ad592c24bbede31a28c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab90cb3957da44618c624bcdd8f063f4"
I20260619 14:53:40.597709 17145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97969f26a258425b894518c24cfe420b" candidate_uuid: "d0209daabca44ad592c24bbede31a28c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2571b6cc039e4835898ee13d156e2225"
I20260619 14:53:40.597810 17408 raft_consensus.cc:3037] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:53:40.597955 17145 raft_consensus.cc:3037] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:53:40.600490 17408 raft_consensus.cc:2454] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0209daabca44ad592c24bbede31a28c in term 1.
I20260619 14:53:40.600584 17145 raft_consensus.cc:2454] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0209daabca44ad592c24bbede31a28c in term 1.
I20260619 14:53:40.600881 17211 leader_election.cc:302] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab90cb3957da44618c624bcdd8f063f4, d0209daabca44ad592c24bbede31a28c; no voters:
I20260619 14:53:40.601166 17476 raft_consensus.cc:2781] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:53:40.601644 17476 raft_consensus.cc:686] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 1 LEADER]: Becoming Leader. State: Replica: d0209daabca44ad592c24bbede31a28c, State: Running, Role: LEADER
I20260619 14:53:40.602116 17476 consensus_queue.cc:237] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } } peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } }
I20260619 14:53:40.606943 17004 catalog_manager.cc:5495] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c reported cstate change: term changed from 0 to 1, leader changed from <none> to d0209daabca44ad592c24bbede31a28c (127.0.241.2). New cstate: current_term: 1 leader_uuid: "d0209daabca44ad592c24bbede31a28c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0209daabca44ad592c24bbede31a28c" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 38959 } health_report { overall_health: HEALTHY } } }
W20260619 14:53:40.618466 17323 tablet.cc:2369] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:53:40.657727 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.651962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.0.241.2:38959)
0619 14:53:40.652217 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.652224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.652317 (+ 93us) server_negotiation.cc:407] Connection header received
0619 14:53:40.652561 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.652565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.652672 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.652827 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:53:40.653215 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.653968 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.655298 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.655838 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.655981 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.656508 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:53:40.657046 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:53:40.657213 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:40.657367 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 14:53:40.657450 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:53:40.665880 17145 raft_consensus.cc:1261] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225 [term 1 FOLLOWER]: Refusing update from remote peer d0209daabca44ad592c24bbede31a28c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:53:40.666023 17408 raft_consensus.cc:1261] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4 [term 1 FOLLOWER]: Refusing update from remote peer d0209daabca44ad592c24bbede31a28c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:53:40.666970 17480 consensus_queue.cc:1035] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2571b6cc039e4835898ee13d156e2225" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:53:40.667292 17476 consensus_queue.cc:1035] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab90cb3957da44618c624bcdd8f063f4" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 45875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:53:40.674255 17191 tablet.cc:2369] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:53:40.691656 17005 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:53:40.691921 17005 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:53:40.695152 17005 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Sending DeleteTablet for 3 replicas of tablet 97969f26a258425b894518c24cfe420b
I20260619 14:53:40.696257 17388 tablet_service.cc:1514] Processing DeleteTablet for tablet 97969f26a258425b894518c24cfe420b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:53:40 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:39960
I20260619 14:53:40.696377 17125 tablet_service.cc:1514] Processing DeleteTablet for tablet 97969f26a258425b894518c24cfe420b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:53:40 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:43150
I20260619 14:53:40.696421 17257 tablet_service.cc:1514] Processing DeleteTablet for tablet 97969f26a258425b894518c24cfe420b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:53:40 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:55600
I20260619 14:53:40.697122 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 17042
I20260619 14:53:40.698793 17494 tablet_replica.cc:330] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225: stopping tablet replica
I20260619 14:53:40.699352 17494 raft_consensus.cc:2229] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:53:40.700065 17494 raft_consensus.cc:2258] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:53:40.700980 17495 tablet_replica.cc:330] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c: stopping tablet replica
I20260619 14:53:40.701503 17495 raft_consensus.cc:2229] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:53:40.702277 17495 raft_consensus.cc:2258] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:53:40.702389 17494 ts_tablet_manager.cc:1905] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:53:40.704777 17495 ts_tablet_manager.cc:1905] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:53:40.709918 17493 tablet_replica.cc:330] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4: stopping tablet replica
I20260619 14:53:40.710328 17493 raft_consensus.cc:2229] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:53:40.710862 17493 raft_consensus.cc:2258] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:53:40.714933 17495 ts_tablet_manager.cc:1918] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:53:40.715132 17495 log.cc:1196] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/97969f26a258425b894518c24cfe420b
I20260619 14:53:40.715555 17494 ts_tablet_manager.cc:1918] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:53:40.715764 17495 ts_tablet_manager.cc:1939] T 97969f26a258425b894518c24cfe420b P d0209daabca44ad592c24bbede31a28c: Deleting consensus metadata
I20260619 14:53:40.715797 17494 log.cc:1196] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/97969f26a258425b894518c24cfe420b
I20260619 14:53:40.716351 17494 ts_tablet_manager.cc:1939] T 97969f26a258425b894518c24cfe420b P 2571b6cc039e4835898ee13d156e2225: Deleting consensus metadata
I20260619 14:53:40.716782 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.698210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.0.241.1:45575)
0619 14:53:40.698825 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.698832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.698850 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:53:40.698902 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.698907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.699002 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.699096 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:40.701054 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.702149 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.704121 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.704746 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.704868 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.708678 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:40.708689 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:40.708703 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:40.708756 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:40.712791 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:40.713730 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:40.713739 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:40.713743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:40.713925 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:40.714387 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:40.714392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:40.714396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:40.714874 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:40.715379 (+ 505us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:40.716133 (+ 754us) server_negotiation.cc:299] Negotiation successful
0619 14:53:40.716377 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490}
I20260619 14:53:40.717778 17493 ts_tablet_manager.cc:1905] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:53:40.717765 16989 catalog_manager.cc:4841] TS d0209daabca44ad592c24bbede31a28c (127.0.241.2:38959): tablet 97969f26a258425b894518c24cfe420b (table test-table [id=7e4e9ae782f84082a4de20eff9d91823]) successfully deleted
I20260619 14:53:40.718169 17185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:53:40.718879 16992 catalog_manager.cc:4841] TS 2571b6cc039e4835898ee13d156e2225 (127.0.241.1:45575): tablet 97969f26a258425b894518c24cfe420b (table test-table [id=7e4e9ae782f84082a4de20eff9d91823]) successfully deleted
I20260619 14:53:40.725891 17493 ts_tablet_manager.cc:1918] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:53:40.726053 17493 log.cc:1196] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/97969f26a258425b894518c24cfe420b
I20260619 14:53:40.726559 17493 ts_tablet_manager.cc:1939] T 97969f26a258425b894518c24cfe420b P ab90cb3957da44618c624bcdd8f063f4: Deleting consensus metadata
I20260619 14:53:40.728339 16990 catalog_manager.cc:4841] TS ab90cb3957da44618c624bcdd8f063f4 (127.0.241.3:45875): tablet 97969f26a258425b894518c24cfe420b (table test-table [id=7e4e9ae782f84082a4de20eff9d91823]) successfully deleted
I20260619 14:53:40.810179 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 17042
I20260619 14:53:40.828338 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 17194
I20260619 14:53:40.842654 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.829129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55626 (local address 127.0.241.2:38959)
0619 14:53:40.829317 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.829325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.829345 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:53:40.829556 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.829562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.829641 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.829749 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:40.831200 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.832151 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.833533 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.833998 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.834136 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.836724 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:40.836740 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:40.836754 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:40.836793 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:40.839888 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:40.840621 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:40.840628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:40.840630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:40.840758 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:40.841133 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:40.841139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:40.841142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:40.841501 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:40.841869 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:40.842073 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 14:53:40.842305 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:53:40.843334 17317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:53:40.956084 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 17194
I20260619 14:53:40.975955 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 17325
I20260619 14:53:40.988667 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:40.976635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.0.241.3:45875)
0619 14:53:40.976840 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:40.976844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:40.976864 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:53:40.977004 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:40.977008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:40.977073 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:40.977154 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:40.978401 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.979103 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.980506 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:40.980887 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:40.980977 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:40.983557 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:40.983569 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:40.983584 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:40.983614 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:40.986282 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:40.986861 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:40.986868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:40.986870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:40.987005 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:40.987302 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:40.987305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:40.987307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:40.987702 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:40.987914 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:40.988162 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 14:53:40.988306 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 14:53:40.989318 17448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:53:41.079232 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 17325
I20260619 14:53:41.095585 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 16972
I20260619 14:53:41.096740 17035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:53:41.211751 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 16972
I20260619 14:54:01.233619 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:37673
--webserver_interface=127.0.241.62
--webserver_port=40399
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:01.560307 17500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:01.560652 17500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:01.560819 17500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:01.569236 17500 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:54:01.569358 17500 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:54:01.569446 17500 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:54:01.569523 17500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:01.569610 17500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:01.569708 17500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:01.569774 17500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:01.569854 17500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:01.582840 17500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:37673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=40399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:01.584189 17500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:01.585505 17500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:01.597721 17508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:01.597755 17505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:01.597802 17506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:01.600844 17500 server_base.cc:1034] running on GCE node
I20260619 14:54:01.601598 17500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:01.602957 17500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:01.604214 17500 hybrid_clock.cc:648] HybridClock initialized: now 1781880841604138 us; error 56 us; skew 500 ppm
I20260619 14:54:01.604580 17500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:01 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:01.610832 17500 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:54:01.612484 17500 webserver.cc:460] Webserver started at http://127.0.241.62:40399/ using document root <none> and password file <none>
I20260619 14:54:01.613241 17500 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:01.613332 17500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:01.618737 17500 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:54:01.621615 17515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:01.622910 17500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:54:01.623109 17500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "06789fbb3e9748fa8a144c8085bcf9fc"
format_stamp: "Formatted at 2026-06-19 14:53:15 on dist-test-slave-p3ml"
server_key: "746e7862ecc6743709246d351510fecd"
server_key_iv: "c79af1467fd59802820335394b50d306"
server_key_version: "encryptionkey@0"
I20260619 14:54:01.624130 17500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:01.637498 17500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:01.641271 17500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:01.641551 17500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:01.659934 17500 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:37673
I20260619 14:54:01.659960 17566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:37673 every 8 connection(s)
I20260619 14:54:01.661409 17500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:01.665733 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 17500
I20260619 14:54:01.666574 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:45575
--local_ip_for_outbound_sockets=127.0.241.1
--tserver_master_addrs=127.0.241.62:37673
--webserver_port=34271
--webserver_interface=127.0.241.1
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:54:01.668227 17567 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:54:01.671324 17567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Bootstrap starting.
I20260619 14:54:01.691007 17567 log.cc:824] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:01.706741 17567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:54:01.707288 17567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Bootstrap complete.
I20260619 14:54:01.716653 17567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } }
I20260619 14:54:01.717543 17567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06789fbb3e9748fa8a144c8085bcf9fc, State: Initialized, Role: FOLLOWER
I20260619 14:54:01.718190 17567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } }
I20260619 14:54:01.718428 17567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:01.718554 17567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:01.718743 17567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:54:01.722435 17567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } }
I20260619 14:54:01.722911 17567 leader_election.cc:302] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06789fbb3e9748fa8a144c8085bcf9fc; no voters:
I20260619 14:54:01.723598 17567 leader_election.cc:288] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:54:01.723750 17571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:54:01.724933 17571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [term 3 LEADER]: Becoming Leader. State: Replica: 06789fbb3e9748fa8a144c8085bcf9fc, State: Running, Role: LEADER
I20260619 14:54:01.725495 17571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } }
I20260619 14:54:01.725805 17567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:01.727983 17573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06789fbb3e9748fa8a144c8085bcf9fc. Latest consensus state: current_term: 3 leader_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } } }
I20260619 14:54:01.728318 17573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:01.728744 17572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06789fbb3e9748fa8a144c8085bcf9fc" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 37673 } } }
I20260619 14:54:01.729004 17572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:01.738343 17576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:01.748757 17576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:01.750698 17576 catalog_manager.cc:1259] Loaded cluster ID: 20d029393d87406a9d6c0c59d030882f
I20260619 14:54:01.750823 17576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:01.756697 17576 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:01.758879 17576 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Loaded TSK: 0
I20260619 14:54:01.759052 17576 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 06789fbb3e9748fa8a144c8085bcf9fc: Loaded TSK: 1
F20260619 14:54:01.760490 17576 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f381003e9ad google::LogMessage::Fail() at ??:0
@ 0x7f381004188a google::LogMessage::SendToLog() at ??:0
@ 0x7f381003e4de google::LogMessage::Flush() at ??:0
@ 0x7f381003fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3813da27d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3813d9173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f3825a6d1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3825a6c783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3825af82a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f3825af80c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f3825b425e9 std::function<>::operator()() at ??:0
@ 0x7f3825a70cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3825a6ec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3825af6610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3825af6439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3812c8a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3812c8d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3812c8d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3812c68eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3815f426db start_thread at ??:0
@ 0x7f380eed861f clone at ??:0
W20260619 14:54:02.070619 17569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:02.070895 17569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:02.070987 17569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:02.079017 17569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:02.079120 17569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:02.079233 17569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:54:02.089955 17569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:45575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=34271
--tserver_master_addrs=127.0.241.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:02.091542 17569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:02.092800 17569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:02.105648 17614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:02.105705 17617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:02.105655 17615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:02.106680 17569 server_base.cc:1034] running on GCE node
I20260619 14:54:02.107353 17569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:02.108309 17569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:02.109535 17569 hybrid_clock.cc:648] HybridClock initialized: now 1781880842109494 us; error 55 us; skew 500 ppm
I20260619 14:54:02.109856 17569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:02 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:02.113229 17569 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:54:02.114712 17569 webserver.cc:460] Webserver started at http://127.0.241.1:34271/ using document root <none> and password file <none>
I20260619 14:54:02.115406 17569 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:02.115618 17569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:02.121145 17569 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:54:02.124028 17624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:02.125375 17569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:54:02.125558 17569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "2571b6cc039e4835898ee13d156e2225"
format_stamp: "Formatted at 2026-06-19 14:53:16 on dist-test-slave-p3ml"
server_key: "c8e6988fd8f967c1f209ce155e1dc595"
server_key_iv: "e087c6f6e12722a4f6665131c777e0f5"
server_key_version: "encryptionkey@0"
I20260619 14:54:02.126535 17569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:02.146456 17569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:02.150743 17569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:02.151033 17569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:02.152098 17569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:02.153470 17569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:02.153597 17569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:02.153745 17569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:02.153838 17569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:02.188683 17569 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:45575
I20260619 14:54:02.188771 17736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:45575 every 8 connection(s)
I20260619 14:54:02.190138 17569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:54:02.190949 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 17569
I20260619 14:54:02.191787 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:38959
--local_ip_for_outbound_sockets=127.0.241.2
--tserver_master_addrs=127.0.241.62:37673
--webserver_port=42615
--webserver_interface=127.0.241.2
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:54:02.195804 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:02.193399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:34989)
0619 14:54:02.194310 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:02.194591 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":332,"threads_started":1}
W20260619 14:54:02.196594 17737 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:37673 (0 consecutive failures): Network error: Failed to ping master at 127.0.241.62:37673: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
I20260619 14:54:02.197574 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:02.197115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:44423)
0619 14:54:02.197311 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:02.197408 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 14:54:02.198572 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:02.198137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:51599)
0619 14:54:02.198307 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:02.198410 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:54:02.198832 17737 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:54:02.516147 17740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:02.516476 17740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:02.516654 17740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:02.524083 17740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:02.524252 17740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:02.524461 17740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:54:02.534824 17740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:38959
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=42615
--tserver_master_addrs=127.0.241.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:02.536105 17740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:02.537381 17740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:02.550153 17745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:02.550194 17746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:02.550315 17748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:02.550981 17740 server_base.cc:1034] running on GCE node
I20260619 14:54:02.551604 17740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:02.552459 17740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:02.553685 17740 hybrid_clock.cc:648] HybridClock initialized: now 1781880842553631 us; error 39 us; skew 500 ppm
I20260619 14:54:02.554035 17740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:02 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:02.557300 17740 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:54:02.558854 17740 webserver.cc:460] Webserver started at http://127.0.241.2:42615/ using document root <none> and password file <none>
I20260619 14:54:02.559626 17740 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:02.559762 17740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:02.565196 17740 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:54:02.567989 17755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:02.569285 17740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:54:02.569482 17740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "d0209daabca44ad592c24bbede31a28c"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-p3ml"
server_key: "abe4a32fa5d55789b7ebfbca799a605d"
server_key_iv: "34d762ba63d649700d113e2db184d291"
server_key_version: "encryptionkey@0"
I20260619 14:54:02.570430 17740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:02.591899 17740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:02.595921 17740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:02.596189 17740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:02.597257 17740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:02.598618 17740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:02.598702 17740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:02.598857 17740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:02.598954 17740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:02.632082 17740 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:38959
I20260619 14:54:02.632164 17867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:38959 every 8 connection(s)
I20260619 14:54:02.633579 17740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:54:02.636051 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 17740
I20260619 14:54:02.636696 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:45875
--local_ip_for_outbound_sockets=127.0.241.3
--tserver_master_addrs=127.0.241.62:37673
--webserver_port=33073
--webserver_interface=127.0.241.3
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:54:02.638932 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:02.636571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:33081)
0619 14:54:02.637487 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:02.637770 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":393,"threads_started":1}
W20260619 14:54:02.639611 17868 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:37673 (0 consecutive failures): Network error: Failed to ping master at 127.0.241.62:37673: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
I20260619 14:54:02.640520 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:02.640093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:55713)
0619 14:54:02.640278 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:02.640371 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:54:02.641505 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:02.641077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:50589)
0619 14:54:02.641253 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:02.641347 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 14:54:02.641805 17868 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:54:02.966980 17871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:02.967254 17871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:02.967456 17871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:02.974831 17871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:02.974958 17871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:02.975148 17871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:54:02.984724 17871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:45875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=33073
--tserver_master_addrs=127.0.241.62:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:02.985977 17871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:02.987284 17871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:03.000054 17876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:03.000088 17879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:02.999971 17877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:03.001109 17871 server_base.cc:1034] running on GCE node
I20260619 14:54:03.001714 17871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:03.002629 17871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:03.003878 17871 hybrid_clock.cc:648] HybridClock initialized: now 1781880843003812 us; error 51 us; skew 500 ppm
I20260619 14:54:03.004293 17871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:03 dist-test-slave-p3ml krb5kdc[16420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:03.007400 17871 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:54:03.008857 17871 webserver.cc:460] Webserver started at http://127.0.241.3:33073/ using document root <none> and password file <none>
I20260619 14:54:03.009518 17871 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:03.009665 17871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:03.015125 17871 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 14:54:03.017733 17886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:03.018982 17871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:54:03.019160 17871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "ab90cb3957da44618c624bcdd8f063f4"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-p3ml"
server_key: "0f4bfe03c4aaf107313172944c344dca"
server_key_iv: "4ca4326ffff64e421eeef24a12aa8d00"
server_key_version: "encryptionkey@0"
I20260619 14:54:03.020205 17871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:03.039292 17871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:03.043532 17871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:03.043818 17871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:03.044858 17871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:03.046185 17871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:03.046307 17871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:03.046443 17871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:03.046537 17871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:03.080036 17871 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:45875
I20260619 14:54:03.080125 17998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:45875 every 8 connection(s)
I20260619 14:54:03.081462 17871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:54:03.085846 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 17871
I20260619 14:54:03.086211 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:03.084391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:35271)
0619 14:54:03.085110 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:03.085314 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":279,"threads_started":1}
W20260619 14:54:03.086970 17999 heartbeater.cc:644] Failed to heartbeat to 127.0.241.62:37673 (0 consecutive failures): Network error: Failed to ping master at 127.0.241.62:37673: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
I20260619 14:54:03.087921 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:03.087548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:51793)
0619 14:54:03.087722 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:03.087818 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:54:03.088718 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:03.088392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:55223)
0619 14:54:03.088530 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:03.088611 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 14:54:03.088933 17999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:54:03.200169 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:03.199338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:56493)
0619 14:54:03.199832 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:03.199932 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":270,"threads_started":1}
I20260619 14:54:03.642930 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:03.642268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:46795)
0619 14:54:03.642675 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:03.642750 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":208,"threads_started":1}
I20260619 14:54:04.090225 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.089426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:47665)
0619 14:54:04.089890 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.089993 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":235,"threads_started":1}
I20260619 14:54:04.203512 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.200990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:39525)
0619 14:54:04.203132 (+ 2142us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.203250 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1966,"thread_start_us":1858,"threads_started":1}
I20260619 14:54:04.644536 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.643830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:44031)
0619 14:54:04.644278 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.644361 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":211,"threads_started":1}
I20260619 14:54:05.091992 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:05.091266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:34083)
0619 14:54:05.091760 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:05.091840 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":263,"threads_started":1}
I20260619 14:54:05.205084 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:05.204350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:41811)
0619 14:54:05.204842 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:05.204925 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":247,"threads_started":1}
I20260619 14:54:05.646209 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:05.645481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:41167)
0619 14:54:05.645932 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:05.646016 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":234,"threads_started":1}
I20260619 14:54:06.093461 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:06.092769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:34679)
0619 14:54:06.093230 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:06.093311 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":241,"threads_started":1}
I20260619 14:54:06.206688 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:06.205916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:45713)
0619 14:54:06.206383 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:06.206487 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":229,"threads_started":1}
I20260619 14:54:06.647979 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:06.647189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:47637)
0619 14:54:06.647731 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:06.647812 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":295,"threads_started":1}
I20260619 14:54:07.095124 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:07.094411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:55015)
0619 14:54:07.094868 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:07.094946 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":244,"threads_started":1}
I20260619 14:54:07.208400 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:07.207628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:45767)
0619 14:54:07.208108 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:07.208215 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":234,"threads_started":1}
I20260619 14:54:07.649498 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:07.648778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:43077)
0619 14:54:07.649238 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:07.649332 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":244,"threads_started":1}
I20260619 14:54:08.096736 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.096041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:50353)
0619 14:54:08.096493 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:08.096569 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":255,"threads_started":1}
I20260619 14:54:08.210090 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.209308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:45463)
0619 14:54:08.209838 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:08.209924 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":358,"threads_started":1}
I20260619 14:54:08.651206 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.650400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:39299)
0619 14:54:08.650967 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:08.651048 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":351,"threads_started":1}
I20260619 14:54:09.098363 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.097659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:44431)
0619 14:54:09.098121 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:09.098206 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":243,"threads_started":1}
I20260619 14:54:09.211936 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.210955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:53667)
0619 14:54:09.211499 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:09.211620 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":212,"threads_started":1}
I20260619 14:54:09.652881 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.652075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:43265)
0619 14:54:09.652562 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:09.652661 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":205,"threads_started":1}
I20260619 14:54:10.100006 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.099206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:38861)
0619 14:54:10.099725 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.099802 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":258,"threads_started":1}
I20260619 14:54:10.213497 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.212764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:51387)
0619 14:54:10.213260 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.213337 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":268,"threads_started":1}
I20260619 14:54:10.654572 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.653846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:37945)
0619 14:54:10.654342 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.654419 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":264,"threads_started":1}
I20260619 14:54:11.101523 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:11.100854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:51171)
0619 14:54:11.101289 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:11.101367 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":200,"threads_started":1}
I20260619 14:54:11.215052 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:11.214317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:57747)
0619 14:54:11.214792 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:11.214873 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":237,"threads_started":1}
I20260619 14:54:11.656118 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:11.655379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:46839)
0619 14:54:11.655866 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:11.655941 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":279,"threads_started":1}
I20260619 14:54:12.103160 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:12.102443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:50437)
0619 14:54:12.102910 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:12.102997 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":246,"threads_started":1}
I20260619 14:54:12.216969 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:12.215951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:58503)
0619 14:54:12.216469 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:12.216573 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":266,"threads_started":1}
I20260619 14:54:12.657693 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:12.657000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:56071)
0619 14:54:12.657463 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:12.657537 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":262,"threads_started":1}
I20260619 14:54:13.104751 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:13.104050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:58197)
0619 14:54:13.104504 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:13.104581 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":252,"threads_started":1}
I20260619 14:54:13.218737 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:13.218019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:37951)
0619 14:54:13.218460 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:13.218551 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
I20260619 14:54:13.659276 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:13.658551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:46561)
0619 14:54:13.659010 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:13.659089 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260619 14:54:14.106299 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:14.105609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:47355)
0619 14:54:14.106027 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:14.106103 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":216,"threads_started":1}
I20260619 14:54:14.220695 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:14.219829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:46917)
0619 14:54:14.220322 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:14.220438 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":220,"threads_started":1}
I20260619 14:54:14.660938 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:14.660234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:52099)
0619 14:54:14.660705 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:14.660778 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":285,"threads_started":1}
I20260619 14:54:15.108116 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:15.107342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:53185)
0619 14:54:15.107843 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:15.107925 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":287,"threads_started":1}
I20260619 14:54:15.222316 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:15.221586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:37411)
0619 14:54:15.222033 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:15.222135 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":226,"threads_started":1}
I20260619 14:54:15.662436 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:15.661747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:53647)
0619 14:54:15.662177 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:15.662273 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":212,"threads_started":1}
I20260619 14:54:16.109711 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:16.109000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:57589)
0619 14:54:16.109429 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:16.109536 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":213,"threads_started":1}
I20260619 14:54:16.224092 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:16.223342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:36137)
0619 14:54:16.223824 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:16.223915 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":267,"threads_started":1}
I20260619 14:54:16.664276 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:16.663500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:60867)
0619 14:54:16.664022 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:16.664101 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":268,"threads_started":1}
I20260619 14:54:17.111340 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:17.110650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:37865)
0619 14:54:17.111086 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:17.111161 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":243,"threads_started":1}
I20260619 14:54:17.226169 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:17.225290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.1:33869)
0619 14:54:17.225886 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:17.225986 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":380,"threads_started":1}
I20260619 14:54:17.666121 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:17.665434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.2:58497)
0619 14:54:17.665845 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:17.665955 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":203,"threads_started":1}
I20260619 14:54:18.095598 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 17569
I20260619 14:54:18.113214 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:18.112438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:37673 (local address 127.0.241.3:35381)
0619 14:54:18.112922 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:18.113045 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.62:37673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":253,"threads_started":1}
I20260619 14:54:18.113320 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:18.096607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.0.241.1:45575)
0619 14:54:18.097308 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:18.097329 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:18.097663 (+ 334us) server_negotiation.cc:407] Connection header received
0619 14:54:18.099513 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:18.099522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:18.099953 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:18.100243 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:18.101650 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:18.102833 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:18.104177 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:18.104803 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:18.104942 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:18.107379 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:18.107386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:18.107418 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:18.107514 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:18.110162 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:18.110751 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:18.110757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:18.110763 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:18.110931 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:18.111235 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:18.111238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:18.111240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:18.111842 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:18.112164 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:18.112606 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 14:54:18.113009 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":572,"threads_started":1}
I20260619 14:54:18.114693 17732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:18.199247 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 17569
I20260619 14:54:18.215865 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 17740
I20260619 14:54:18.234110 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:18.216838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.0.241.2:38959)
0619 14:54:18.217407 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:18.217419 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:18.217456 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:54:18.219092 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:18.219101 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:18.219602 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:18.219908 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:18.221166 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:18.222416 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:18.223964 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:18.224727 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:18.224971 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:18.227648 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:18.227655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:18.227670 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:18.227708 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:18.230847 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:18.231717 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:18.231723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:18.231729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:18.231908 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:18.232300 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:18.232304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:18.232306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:18.232813 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:18.233039 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:18.233406 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 14:54:18.233816 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":230,"threads_started":1}
I20260619 14:54:18.235507 17863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:18.317754 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 17740
I20260619 14:54:18.335366 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 17871
I20260619 14:54:18.353101 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:18.336453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.0.241.3:45875)
0619 14:54:18.337054 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:18.337066 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:18.337105 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:54:18.338706 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:18.338715 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:18.339112 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:18.339507 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:18.340951 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:18.342179 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:18.343580 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:18.344258 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:18.344471 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:18.347064 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:18.347072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:18.347085 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:18.347120 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:18.349829 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:18.350503 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:18.350510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:18.350516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:18.350678 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:18.351061 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:18.351068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:18.351071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:18.351654 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:18.351962 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:18.352416 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 14:54:18.352811 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":279,"threads_started":1}
I20260619 14:54:18.354453 17994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:18.436829 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 17871
2026-06-19T14:54:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64200 ms)
[----------] 30 tests from SecurityITest (293118 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:18 dist-test-slave-p3ml krb5kdc[18056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:18 dist-test-slave-p3ml krb5kdc[18056](info): set up 2 sockets
Jun 19 14:54:18 dist-test-slave-p3ml krb5kdc[18056](info): commencing operation
krb5kdc: starting...
W20260619 14:54:19.551687 964 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:19 dist-test-slave-p3ml krb5kdc[18056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:54:19.786386 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:42731
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:42731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:20.108803 18069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:20.109143 18069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:20.109311 18069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:20.116526 18069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:20.116676 18069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:20.116789 18069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:20.116894 18069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:20.116990 18069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:20.126583 18069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:42731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:42731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:20.127942 18069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:20.129282 18069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:20.141423 18075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:20.141455 18074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:20.141963 18077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:20.142567 18069 server_base.cc:1034] running on GCE node
I20260619 14:54:20.143285 18069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:20.144598 18069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:20.145787 18069 hybrid_clock.cc:648] HybridClock initialized: now 1781880860145729 us; error 51 us; skew 500 ppm
I20260619 14:54:20.146121 18069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:20 dist-test-slave-p3ml krb5kdc[18056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:20.153493 18069 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:54:20.155432 18069 webserver.cc:460] Webserver started at http://127.0.241.62:34135/ using document root <none> and password file <none>
I20260619 14:54:20.156275 18069 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:20.156410 18069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:20.156713 18069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:20.159727 18069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "1b7db1b25e88492ca7b31f7aa1c9808c"
format_stamp: "Formatted at 2026-06-19 14:54:20 on dist-test-slave-p3ml"
server_key: "e0b3a82a6288dd1be283320bdd934bce"
server_key_iv: "334524c4f2c51df151bab29d0fccbe06"
server_key_version: "encryptionkey@0"
I20260619 14:54:20.160483 18069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "1b7db1b25e88492ca7b31f7aa1c9808c"
format_stamp: "Formatted at 2026-06-19 14:54:20 on dist-test-slave-p3ml"
server_key: "e0b3a82a6288dd1be283320bdd934bce"
server_key_iv: "334524c4f2c51df151bab29d0fccbe06"
server_key_version: "encryptionkey@0"
I20260619 14:54:20.165390 18069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:54:20.168702 18084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:20.169922 18069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:54:20.170094 18069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "1b7db1b25e88492ca7b31f7aa1c9808c"
format_stamp: "Formatted at 2026-06-19 14:54:20 on dist-test-slave-p3ml"
server_key: "e0b3a82a6288dd1be283320bdd934bce"
server_key_iv: "334524c4f2c51df151bab29d0fccbe06"
server_key_version: "encryptionkey@0"
I20260619 14:54:20.170287 18069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:20.195140 18069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:20.198952 18069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:20.199218 18069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:20.217088 18069 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:42731
I20260619 14:54:20.217125 18135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:42731 every 8 connection(s)
I20260619 14:54:20.218355 18069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:20.219594 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 18069
I20260619 14:54:20.219800 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:20.220656 964 external_mini_cluster.cc:1428] Setting key ca99820048a2f731c8a91821f7b961e4
I20260619 14:54:20.223081 18136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 14:54:20 dist-test-slave-p3ml krb5kdc[18056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:54:20.235872 18136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c: Bootstrap starting.
I20260619 14:54:20.240010 18136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:20.241226 18136 log.cc:824] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:20.244518 18136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c: No bootstrap required, opened a new log
I20260619 14:54:20.244807 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:20.222646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41158 (local address 127.0.241.62:42731)
0619 14:54:20.223406 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:20.223463 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:20.223519 (+ 56us) server_negotiation.cc:407] Connection header received
0619 14:54:20.225761 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:20.225785 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:20.226259 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:20.226700 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:20.227978 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:20.229988 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:20.231261 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:20.232294 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:20.232491 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:20.236462 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:20.236472 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:20.236489 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:20.236535 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:20.240589 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:20.241235 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:20.241246 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:20.241255 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:20.241468 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:20.241884 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:20.241888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:20.241893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:20.242437 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:20.242766 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:20.243106 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 14:54:20.243544 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":328,"threads_started":1}
I20260619 14:54:20.252655 18136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7db1b25e88492ca7b31f7aa1c9808c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 42731 } }
I20260619 14:54:20.252985 18136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:20.253054 18136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7db1b25e88492ca7b31f7aa1c9808c, State: Initialized, Role: FOLLOWER
I20260619 14:54:20.253571 18136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7db1b25e88492ca7b31f7aa1c9808c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 42731 } }
I20260619 14:54:20.253784 18136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:20.253877 18136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:20.254038 18136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:20.256268 18136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7db1b25e88492ca7b31f7aa1c9808c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 42731 } }
I20260619 14:54:20.256772 18136 leader_election.cc:302] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b7db1b25e88492ca7b31f7aa1c9808c; no voters:
I20260619 14:54:20.257347 18136 leader_election.cc:288] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:20.257503 18141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:20.258244 18141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [term 1 LEADER]: Becoming Leader. State: Replica: 1b7db1b25e88492ca7b31f7aa1c9808c, State: Running, Role: LEADER
I20260619 14:54:20.258785 18141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7db1b25e88492ca7b31f7aa1c9808c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 42731 } }
I20260619 14:54:20.259487 18136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:20.261407 18142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b7db1b25e88492ca7b31f7aa1c9808c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7db1b25e88492ca7b31f7aa1c9808c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 42731 } } }
I20260619 14:54:20.261744 18142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:20.263516 18143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b7db1b25e88492ca7b31f7aa1c9808c. Latest consensus state: current_term: 1 leader_uuid: "1b7db1b25e88492ca7b31f7aa1c9808c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b7db1b25e88492ca7b31f7aa1c9808c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 42731 } } }
I20260619 14:54:20.263794 18143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:20.268440 18148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:20.273656 18148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:20.281965 18148 catalog_manager.cc:1347] Generated new cluster ID: f05da7ec02cc4d038f8114b6e6213463
I20260619 14:54:20.282065 18148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:20.295208 18148 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:20.296499 18148 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:20.302697 18148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c: Generated new TSK 0
I20260619 14:54:20.303498 18148 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:54:20.375066 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:42731
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:20.708330 18162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:20.708585 18162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:20.708761 18162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:20.715507 18162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:20.715623 18162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:20.715821 18162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:54:20.724912 18162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:42731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:20.726145 18162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:20.727398 18162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:20.740427 18167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:20.740375 18170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:20.740476 18168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:20.741204 18162 server_base.cc:1034] running on GCE node
I20260619 14:54:20.741758 18162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:20.742727 18162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:20.743921 18162 hybrid_clock.cc:648] HybridClock initialized: now 1781880860743884 us; error 25 us; skew 500 ppm
I20260619 14:54:20.744234 18162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:20 dist-test-slave-p3ml krb5kdc[18056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:20.747256 18162 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:54:20.748585 18162 webserver.cc:460] Webserver started at http://127.0.241.1:43217/ using document root <none> and password file <none>
I20260619 14:54:20.749176 18162 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:20.749270 18162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:20.749521 18162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:20.752336 18162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "3fbf8e22f8b04ff6ba02b815abe353b5"
format_stamp: "Formatted at 2026-06-19 14:54:20 on dist-test-slave-p3ml"
server_key: "f99df610aef35b7fd8590f0228568591"
server_key_iv: "fc04a53e3864adc0b85b36c14a4ea202"
server_key_version: "encryptionkey@0"
I20260619 14:54:20.753405 18162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "3fbf8e22f8b04ff6ba02b815abe353b5"
format_stamp: "Formatted at 2026-06-19 14:54:20 on dist-test-slave-p3ml"
server_key: "f99df610aef35b7fd8590f0228568591"
server_key_iv: "fc04a53e3864adc0b85b36c14a4ea202"
server_key_version: "encryptionkey@0"
I20260619 14:54:20.758152 18162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 14:54:20.761550 18177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:20.762827 18162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:54:20.762998 18162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "3fbf8e22f8b04ff6ba02b815abe353b5"
format_stamp: "Formatted at 2026-06-19 14:54:20 on dist-test-slave-p3ml"
server_key: "f99df610aef35b7fd8590f0228568591"
server_key_iv: "fc04a53e3864adc0b85b36c14a4ea202"
server_key_version: "encryptionkey@0"
I20260619 14:54:20.763183 18162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:20.801793 18162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:20.806097 18162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:20.806409 18162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:20.807363 18162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:20.808682 18162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:20.808790 18162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:20.808944 18162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:20.809018 18162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:20.841508 18162 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:38879
I20260619 14:54:20.841593 18289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:38879 every 8 connection(s)
I20260619 14:54:20.842767 18162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:54:20.844067 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 18162
I20260619 14:54:20.844286 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:54:20.845120 964 external_mini_cluster.cc:1428] Setting key d3b7dc3a84d97155f2732528027cafbb
Jun 19 14:54:20 dist-test-slave-p3ml krb5kdc[18056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:54:20.866441 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:20.845698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:37091 (local address 127.0.241.62:42731)
0619 14:54:20.846235 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:20.846242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:20.847266 (+ 1024us) server_negotiation.cc:407] Connection header received
0619 14:54:20.850134 (+ 2868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:20.850138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:20.850207 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:20.850300 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:20.852461 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:20.853178 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:20.855721 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:20.856095 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:20.856191 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:20.860408 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:20.860413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:20.860417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:20.860455 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:20.863003 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:20.863984 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:20.863988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:20.863990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:20.864108 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:20.864614 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:20.864617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:20.864619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:20.864876 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:20.864985 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:20.865897 (+ 912us) server_negotiation.cc:299] Negotiation successful
0619 14:54:20.866074 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":244,"threads_started":1}
I20260619 14:54:20.867856 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:20.846287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:42731 (local address 127.0.241.1:37091)
0619 14:54:20.847070 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:20.847126 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:20.849817 (+ 2691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:20.850600 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:20.850614 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:20.851415 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:20.852262 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:20.852289 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:20.853339 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:20.853348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:20.855582 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:20.855600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:20.856325 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:20.856334 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:20.856654 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:20.857463 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:20.857486 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:20.860178 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:20.863141 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:20.863153 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:20.863172 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:20.863864 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:20.864219 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:20.864226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:20.864232 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:20.864497 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:20.865007 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:20.865017 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:20.865481 (+ 464us) client_negotiation.cc:769] Sending connection context
0619 14:54:20.865781 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 14:54:20.866325 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":320,"threads_started":1}
I20260619 14:54:20.869458 18290 heartbeater.cc:347] Connected to a master server at 127.0.241.62:42731
I20260619 14:54:20.869813 18290 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:20.870608 18290 heartbeater.cc:511] Master 127.0.241.62:42731 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
I20260619 14:54:20.872676 18101 ts_manager.cc:194] Registered new tserver with Master: 3fbf8e22f8b04ff6ba02b815abe353b5 (127.0.241.1:38879)
I20260619 14:54:20.874275 18101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:37091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:54:20.901983 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:42731
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:21.225490 18296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:21.225817 18296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:21.225999 18296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:21.233047 18296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:21.233213 18296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:21.233398 18296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:54:21.242939 18296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:42731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:21.244174 18296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:21.245478 18296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:21.258605 18301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:21.258558 18304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:21.258642 18302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:21.259805 18296 server_base.cc:1034] running on GCE node
I20260619 14:54:21.260349 18296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:21.261260 18296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:21.262492 18296 hybrid_clock.cc:648] HybridClock initialized: now 1781880861262451 us; error 32 us; skew 500 ppm
I20260619 14:54:21.262753 18296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:21 dist-test-slave-p3ml krb5kdc[18056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:21.265734 18296 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:54:21.267112 18296 webserver.cc:460] Webserver started at http://127.0.241.2:32771/ using document root <none> and password file <none>
I20260619 14:54:21.267823 18296 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:21.267946 18296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:21.268186 18296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:21.270947 18296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "7a07c89b928c469782b639d45fc01934"
format_stamp: "Formatted at 2026-06-19 14:54:21 on dist-test-slave-p3ml"
server_key: "5e8a34c0b20ff1d0bd50cff284b6cf9c"
server_key_iv: "55cf8426e891adab5610acbbd1a15576"
server_key_version: "encryptionkey@0"
I20260619 14:54:21.271718 18296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "7a07c89b928c469782b639d45fc01934"
format_stamp: "Formatted at 2026-06-19 14:54:21 on dist-test-slave-p3ml"
server_key: "5e8a34c0b20ff1d0bd50cff284b6cf9c"
server_key_iv: "55cf8426e891adab5610acbbd1a15576"
server_key_version: "encryptionkey@0"
I20260619 14:54:21.276484 18296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:54:21.279568 18311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:21.280761 18296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:54:21.280884 18296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "7a07c89b928c469782b639d45fc01934"
format_stamp: "Formatted at 2026-06-19 14:54:21 on dist-test-slave-p3ml"
server_key: "5e8a34c0b20ff1d0bd50cff284b6cf9c"
server_key_iv: "55cf8426e891adab5610acbbd1a15576"
server_key_version: "encryptionkey@0"
I20260619 14:54:21.281039 18296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:21.297813 18296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:21.301960 18296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:21.302246 18296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:21.303202 18296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:21.304379 18296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:21.304446 18296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:21.304556 18296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:21.304601 18296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:21.339561 18296 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:41421
I20260619 14:54:21.339586 18423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:41421 every 8 connection(s)
I20260619 14:54:21.340948 18296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:54:21.343152 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 18296
I20260619 14:54:21.343349 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:54:21.343957 964 external_mini_cluster.cc:1428] Setting key 74a01eea9825dbfa977ae5d8ae9ce5b6
Jun 19 14:54:21 dist-test-slave-p3ml krb5kdc[18056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:54:21.363356 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.343599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:52789 (local address 127.0.241.62:42731)
0619 14:54:21.343773 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:21.343777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:21.345311 (+ 1534us) server_negotiation.cc:407] Connection header received
0619 14:54:21.347389 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:21.347394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:21.347509 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:21.347622 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:21.349077 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.349850 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.351387 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.352029 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.352163 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:21.356272 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:21.356281 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:21.356289 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:21.356334 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:21.360089 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:21.360821 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:21.360829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:21.360835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:21.361001 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:21.361409 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:21.361416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:21.361422 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:21.361825 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:21.361979 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:21.362654 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 14:54:21.362930 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:54:21.363955 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.344266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:42731 (local address 127.0.241.2:52789)
0619 14:54:21.345134 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:21.345183 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:21.347132 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:21.347854 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:21.347862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:21.348349 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:21.348905 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:21.348922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.350020 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.350024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:21.351229 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:21.351241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.352267 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.352275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:21.352510 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:21.353238 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:21.353279 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:21.356029 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:21.360223 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:21.360230 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:21.360241 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:21.360695 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:21.361119 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:21.361122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:21.361124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:21.361293 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:21.361953 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:21.361958 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:21.362250 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 14:54:21.362493 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 14:54:21.362866 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":354,"threads_started":1}
I20260619 14:54:21.365355 18424 heartbeater.cc:347] Connected to a master server at 127.0.241.62:42731
I20260619 14:54:21.365681 18424 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:21.366670 18424 heartbeater.cc:511] Master 127.0.241.62:42731 requested a full tablet report, sending...
I20260619 14:54:21.368355 18101 ts_manager.cc:194] Registered new tserver with Master: 7a07c89b928c469782b639d45fc01934 (127.0.241.2:41421)
I20260619 14:54:21.369354 18101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:52789
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:54:21.398777 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:42731
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:33237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:21.717125 18429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:21.717408 18429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:21.717573 18429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:21.724512 18429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:21.724647 18429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:21.724835 18429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:54:21.733922 18429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:33237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:42731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:21.735133 18429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:21.736413 18429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:21.749794 18435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:21.749868 18434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:21.749806 18437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:21.752874 18429 server_base.cc:1034] running on GCE node
I20260619 14:54:21.753477 18429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:21.754375 18429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:21.755617 18429 hybrid_clock.cc:648] HybridClock initialized: now 1781880861755585 us; error 46 us; skew 500 ppm
I20260619 14:54:21.755926 18429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:21 dist-test-slave-p3ml krb5kdc[18056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:21.759094 18429 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:54:21.760645 18429 webserver.cc:460] Webserver started at http://127.0.241.3:46521/ using document root <none> and password file <none>
I20260619 14:54:21.761384 18429 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:21.761545 18429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:21.761848 18429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:21.764828 18429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "866ac2bcc865460ca0b392cd91aad8f7"
format_stamp: "Formatted at 2026-06-19 14:54:21 on dist-test-slave-p3ml"
server_key: "1c56f0a7240f89d289f22a8fd46f2a7f"
server_key_iv: "d90926186f722e384b05725f177eace1"
server_key_version: "encryptionkey@0"
I20260619 14:54:21.765653 18429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "866ac2bcc865460ca0b392cd91aad8f7"
format_stamp: "Formatted at 2026-06-19 14:54:21 on dist-test-slave-p3ml"
server_key: "1c56f0a7240f89d289f22a8fd46f2a7f"
server_key_iv: "d90926186f722e384b05725f177eace1"
server_key_version: "encryptionkey@0"
I20260619 14:54:21.770982 18429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 14:54:21.774338 18444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:21.775620 18429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:54:21.775790 18429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "866ac2bcc865460ca0b392cd91aad8f7"
format_stamp: "Formatted at 2026-06-19 14:54:21 on dist-test-slave-p3ml"
server_key: "1c56f0a7240f89d289f22a8fd46f2a7f"
server_key_iv: "d90926186f722e384b05725f177eace1"
server_key_version: "encryptionkey@0"
I20260619 14:54:21.775978 18429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:21.805647 18429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:21.809336 18429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:21.809644 18429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:21.810572 18429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:21.811873 18429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:21.811978 18429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:21.812129 18429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:21.812222 18429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:21.844786 18429 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:39589
I20260619 14:54:21.844878 18556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:39589 every 8 connection(s)
I20260619 14:54:21.846206 18429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:54:21.849936 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 18429
I20260619 14:54:21.850133 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:54:21.850677 964 external_mini_cluster.cc:1428] Setting key 367cda8d0e25a3f8a3d800a5fe450055
Jun 19 14:54:21 dist-test-slave-p3ml krb5kdc[18056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880861, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:54:21.866832 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.848696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:55461 (local address 127.0.241.62:42731)
0619 14:54:21.848880 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:21.848885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:21.850099 (+ 1214us) server_negotiation.cc:407] Connection header received
0619 14:54:21.852546 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:21.852554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:21.852693 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:21.852810 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:21.854412 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.855166 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.856766 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.857175 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.857311 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:21.861006 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:21.861011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:21.861014 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:21.861074 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:21.863613 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:21.864344 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:21.864348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:21.864350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:21.864485 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:21.864967 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:21.864971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:21.864973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:21.865328 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:21.865443 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:21.866252 (+ 809us) server_negotiation.cc:299] Negotiation successful
0619 14:54:21.866497 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:54:21.867915 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.849157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:42731 (local address 127.0.241.3:55461)
0619 14:54:21.849934 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:21.849980 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:21.852222 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:21.853063 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:21.853072 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:21.853597 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:21.854244 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:21.854263 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.855319 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.855327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:21.856611 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:21.856622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.857395 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.857405 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:21.857661 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:21.858255 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:21.858276 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:21.860794 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:21.863727 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:21.863733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:21.863743 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:21.864232 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:21.864608 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:21.864616 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:21.864621 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:21.864859 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:21.865466 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:21.865474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:21.865798 (+ 324us) client_negotiation.cc:769] Sending connection context
0619 14:54:21.866111 (+ 313us) client_negotiation.cc:240] Negotiation successful
0619 14:54:21.866654 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":302,"threads_started":1}
I20260619 14:54:21.869148 18557 heartbeater.cc:347] Connected to a master server at 127.0.241.62:42731
I20260619 14:54:21.869447 18557 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:21.870155 18557 heartbeater.cc:511] Master 127.0.241.62:42731 requested a full tablet report, sending...
I20260619 14:54:21.871524 18101 ts_manager.cc:194] Registered new tserver with Master: 866ac2bcc865460ca0b392cd91aad8f7 (127.0.241.3:39589)
I20260619 14:54:21.872303 18101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:55461
I20260619 14:54:21.877661 18290 heartbeater.cc:503] Master 127.0.241.62:42731 was elected leader, sending a full tablet report...
I20260619 14:54:21.878664 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:54:21.895900 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.883288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.0.241.62:42731)
0619 14:54:21.883493 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:21.883501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:21.883897 (+ 396us) server_negotiation.cc:407] Connection header received
0619 14:54:21.884040 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:21.884043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:21.884098 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:21.884179 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:21.885890 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.886603 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.888078 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.888439 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.888556 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:21.891047 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:21.891052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:21.891055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:21.891101 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:21.893536 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:21.894189 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:21.894193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:21.894195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:21.894310 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:21.894651 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:21.894654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:21.894656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:21.894967 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:21.895154 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:21.895485 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 14:54:21.895654 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:54:21.899945 18100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:54:21.907236 18100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:54:21.947551 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.919096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.0.241.1:38879)
0619 14:54:21.937169 (+ 18073us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:21.937180 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:21.937219 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:54:21.937340 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:21.937352 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:21.937683 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:21.937917 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:21.938734 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.941753 (+ 3019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.944787 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.946259 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.946385 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:21.946634 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:21.947010 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 14:54:21.947174 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17884,"thread_start_us":17701,"threads_started":1}
I20260619 14:54:21.947587 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.918947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:38879 (local address 127.0.0.1:58922)
0619 14:54:21.935953 (+ 17006us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:21.935984 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:21.936117 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:21.937991 (+ 1874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:21.937998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:21.938039 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:21.938525 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:21.938543 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.941939 (+ 3396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.941945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:21.944581 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:21.944600 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.946535 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.946542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:21.946750 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:21.946803 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 14:54:21.946939 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 14:54:21.947120 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16843,"mutex_wait_us":71,"thread_start_us":21646,"threads_started":1}
I20260619 14:54:21.948334 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.920644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.0.241.2:41421)
0619 14:54:21.936594 (+ 15950us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:21.936607 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:21.941019 (+ 4412us) server_negotiation.cc:407] Connection header received
0619 14:54:21.941158 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:21.941165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:21.941366 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:21.941562 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:21.942308 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.943624 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.945861 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.947107 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.947217 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:21.947401 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:21.947973 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 14:54:21.948109 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15806,"thread_start_us":15692,"threads_started":1}
I20260619 14:54:21.949066 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.920426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:41421 (local address 127.0.0.1:48054)
0619 14:54:21.940861 (+ 20435us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:21.940880 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:21.941004 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:21.941548 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:21.941553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:21.941577 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:21.942112 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:21.942127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.943782 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.943787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:21.945716 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:21.945728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.947587 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.947593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:21.947754 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:21.947784 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:54:21.948596 (+ 812us) client_negotiation.cc:240] Negotiation successful
0619 14:54:21.948729 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20305,"mutex_wait_us":10,"thread_start_us":15200,"threads_started":1}
I20260619 14:54:21.951385 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.918690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:39589 (local address 127.0.0.1:47634)
0619 14:54:21.933849 (+ 15159us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:21.933911 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:21.934133 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:21.941700 (+ 7567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:21.941708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:21.941732 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:21.942362 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:21.942375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.944419 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.944433 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:21.946672 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:21.946692 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.950581 (+ 3889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.950587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:21.950745 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:21.950787 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 14:54:21.950899 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:54:21.951032 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14841,"mutex_wait_us":72,"thread_start_us":15656,"threads_started":1}
I20260619 14:54:21.951385 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.918895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.0.241.3:39589)
0619 14:54:21.941240 (+ 22345us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:21.941246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:21.941285 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:54:21.941353 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:21.941358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:21.941519 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:21.941704 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:21.942583 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.944252 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.946828 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:21.948605 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:21.950530 (+ 1925us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:21.950743 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:21.950936 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 14:54:21.951080 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22191,"thread_start_us":22176,"threads_started":1}
I20260619 14:54:21.951630 18225 tablet_service.cc:1467] Processing CreateTablet for tablet 4800200b2332400ea38fde5f569ee8d0 (DEFAULT_TABLE table=test-table [id=ac1a11762b8a4ff3ba408c375df06c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:21.952483 18359 tablet_service.cc:1467] Processing CreateTablet for tablet 4800200b2332400ea38fde5f569ee8d0 (DEFAULT_TABLE table=test-table [id=ac1a11762b8a4ff3ba408c375df06c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:21.953125 18225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4800200b2332400ea38fde5f569ee8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:21.953821 18359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4800200b2332400ea38fde5f569ee8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:21.958148 18492 tablet_service.cc:1467] Processing CreateTablet for tablet 4800200b2332400ea38fde5f569ee8d0 (DEFAULT_TABLE table=test-table [id=ac1a11762b8a4ff3ba408c375df06c06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:21.959246 18492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4800200b2332400ea38fde5f569ee8d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:21.967875 18576 tablet_bootstrap.cc:492] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5: Bootstrap starting.
I20260619 14:54:21.968498 18575 tablet_bootstrap.cc:492] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934: Bootstrap starting.
I20260619 14:54:21.970686 18577 tablet_bootstrap.cc:492] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7: Bootstrap starting.
I20260619 14:54:21.973018 18575 tablet_bootstrap.cc:654] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:21.973592 18577 tablet_bootstrap.cc:654] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:21.974345 18575 log.cc:824] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:21.974493 18576 tablet_bootstrap.cc:654] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:21.974670 18577 log.cc:824] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:21.975740 18576 log.cc:824] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:21.977572 18575 tablet_bootstrap.cc:492] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934: No bootstrap required, opened a new log
I20260619 14:54:21.977829 18575 ts_tablet_manager.cc:1397] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 14:54:21.978255 18576 tablet_bootstrap.cc:492] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5: No bootstrap required, opened a new log
I20260619 14:54:21.978488 18576 ts_tablet_manager.cc:1397] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260619 14:54:21.985785 18575 raft_consensus.cc:348] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866ac2bcc865460ca0b392cd91aad8f7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 39589 } } peers { permanent_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38879 } } peers { permanent_uuid: "7a07c89b928c469782b639d45fc01934" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41421 } }
I20260619 14:54:21.985978 18576 raft_consensus.cc:348] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866ac2bcc865460ca0b392cd91aad8f7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 39589 } } peers { permanent_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38879 } } peers { permanent_uuid: "7a07c89b928c469782b639d45fc01934" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41421 } }
I20260619 14:54:21.986255 18575 raft_consensus.cc:374] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:21.986310 18576 raft_consensus.cc:374] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:21.986348 18575 raft_consensus.cc:729] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a07c89b928c469782b639d45fc01934, State: Initialized, Role: FOLLOWER
I20260619 14:54:21.986399 18576 raft_consensus.cc:729] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fbf8e22f8b04ff6ba02b815abe353b5, State: Initialized, Role: FOLLOWER
I20260619 14:54:21.987011 18576 consensus_queue.cc:260] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866ac2bcc865460ca0b392cd91aad8f7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 39589 } } peers { permanent_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38879 } } peers { permanent_uuid: "7a07c89b928c469782b639d45fc01934" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41421 } }
I20260619 14:54:21.986970 18575 consensus_queue.cc:260] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866ac2bcc865460ca0b392cd91aad8f7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 39589 } } peers { permanent_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38879 } } peers { permanent_uuid: "7a07c89b928c469782b639d45fc01934" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41421 } }
I20260619 14:54:21.987620 18577 tablet_bootstrap.cc:492] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7: No bootstrap required, opened a new log
I20260619 14:54:21.987892 18577 ts_tablet_manager.cc:1397] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20260619 14:54:21.988317 18424 heartbeater.cc:503] Master 127.0.241.62:42731 was elected leader, sending a full tablet report...
I20260619 14:54:21.989408 18576 ts_tablet_manager.cc:1428] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260619 14:54:21.990481 18575 ts_tablet_manager.cc:1428] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 14:54:21.996730 18577 raft_consensus.cc:348] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866ac2bcc865460ca0b392cd91aad8f7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 39589 } } peers { permanent_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38879 } } peers { permanent_uuid: "7a07c89b928c469782b639d45fc01934" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41421 } }
I20260619 14:54:21.997110 18577 raft_consensus.cc:374] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:21.997236 18577 raft_consensus.cc:729] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 866ac2bcc865460ca0b392cd91aad8f7, State: Initialized, Role: FOLLOWER
I20260619 14:54:21.997779 18577 consensus_queue.cc:260] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866ac2bcc865460ca0b392cd91aad8f7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 39589 } } peers { permanent_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38879 } } peers { permanent_uuid: "7a07c89b928c469782b639d45fc01934" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41421 } }
I20260619 14:54:21.998690 18557 heartbeater.cc:503] Master 127.0.241.62:42731 was elected leader, sending a full tablet report...
I20260619 14:54:21.999147 18577 ts_tablet_manager.cc:1428] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7: Time spent starting tablet: real 0.011s user 0.001s sys 0.010s
I20260619 14:54:22.055295 18581 raft_consensus.cc:482] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:22.055626 18581 raft_consensus.cc:504] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866ac2bcc865460ca0b392cd91aad8f7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 39589 } } peers { permanent_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38879 } } peers { permanent_uuid: "7a07c89b928c469782b639d45fc01934" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41421 } }
I20260619 14:54:22.057356 18581 leader_election.cc:288] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 866ac2bcc865460ca0b392cd91aad8f7 (127.0.241.3:39589), 7a07c89b928c469782b639d45fc01934 (127.0.241.2:41421)
I20260619 14:54:22.064231 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.057856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:41625 (local address 127.0.241.2:41421)
0619 14:54:22.058068 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:22.058076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:22.058700 (+ 624us) server_negotiation.cc:407] Connection header received
0619 14:54:22.058910 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:22.058916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:22.059001 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:22.059068 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:22.059751 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.060579 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.062462 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.063222 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.063371 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:22.063532 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:22.063829 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 14:54:22.063969 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:54:22.064268 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.057774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:58829 (local address 127.0.241.3:39589)
0619 14:54:22.058006 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:22.058015 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:22.058801 (+ 786us) server_negotiation.cc:407] Connection header received
0619 14:54:22.058977 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:22.058981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:22.059039 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:22.059153 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:22.059907 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.060723 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.062473 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.063232 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.063392 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:22.063525 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:22.063891 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 14:54:22.064032 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 14:54:22.064378 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.057679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:41421 (local address 127.0.241.1:41625)
0619 14:54:22.058589 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:22.058604 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:22.058726 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:22.059121 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:22.059128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:22.059157 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:22.059599 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:22.059609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.060717 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.060722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:22.062311 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:22.062323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.063494 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.063501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:22.063676 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:22.063700 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:54:22.063776 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:54:22.063914 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":3,"thread_start_us":551,"threads_started":1}
I20260619 14:54:22.064419 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.057691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:39589 (local address 127.0.241.1:58829)
0619 14:54:22.058647 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:22.058685 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:22.058834 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:22.059184 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:22.059191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:22.059217 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:22.059723 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:22.059734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.060854 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.060857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:22.062313 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:22.062324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.063547 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.063555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:22.063713 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:22.063734 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:54:22.063844 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 14:54:22.063962 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":106,"thread_start_us":480,"threads_started":1}
I20260619 14:54:22.065332 18379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4800200b2332400ea38fde5f569ee8d0" candidate_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a07c89b928c469782b639d45fc01934" is_pre_election: true
I20260619 14:54:22.065335 18512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4800200b2332400ea38fde5f569ee8d0" candidate_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "866ac2bcc865460ca0b392cd91aad8f7" is_pre_election: true
I20260619 14:54:22.065802 18379 raft_consensus.cc:2454] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fbf8e22f8b04ff6ba02b815abe353b5 in term 0.
I20260619 14:54:22.065817 18512 raft_consensus.cc:2454] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fbf8e22f8b04ff6ba02b815abe353b5 in term 0.
I20260619 14:54:22.066475 18181 leader_election.cc:302] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3fbf8e22f8b04ff6ba02b815abe353b5, 7a07c89b928c469782b639d45fc01934, 866ac2bcc865460ca0b392cd91aad8f7; no voters:
I20260619 14:54:22.066866 18581 raft_consensus.cc:2781] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:54:22.067031 18581 raft_consensus.cc:482] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:54:22.067123 18581 raft_consensus.cc:3037] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:22.069363 18581 raft_consensus.cc:504] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866ac2bcc865460ca0b392cd91aad8f7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 39589 } } peers { permanent_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38879 } } peers { permanent_uuid: "7a07c89b928c469782b639d45fc01934" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41421 } }
I20260619 14:54:22.069942 18581 leader_election.cc:288] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [CANDIDATE]: Term 1 election: Requested vote from peers 866ac2bcc865460ca0b392cd91aad8f7 (127.0.241.3:39589), 7a07c89b928c469782b639d45fc01934 (127.0.241.2:41421)
I20260619 14:54:22.070324 18512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4800200b2332400ea38fde5f569ee8d0" candidate_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "866ac2bcc865460ca0b392cd91aad8f7"
I20260619 14:54:22.070423 18379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4800200b2332400ea38fde5f569ee8d0" candidate_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a07c89b928c469782b639d45fc01934"
I20260619 14:54:22.070560 18512 raft_consensus.cc:3037] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:22.070609 18379 raft_consensus.cc:3037] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:22.072849 18512 raft_consensus.cc:2454] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fbf8e22f8b04ff6ba02b815abe353b5 in term 1.
I20260619 14:54:22.072860 18379 raft_consensus.cc:2454] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fbf8e22f8b04ff6ba02b815abe353b5 in term 1.
I20260619 14:54:22.073258 18179 leader_election.cc:302] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fbf8e22f8b04ff6ba02b815abe353b5, 7a07c89b928c469782b639d45fc01934; no voters:
I20260619 14:54:22.073571 18581 raft_consensus.cc:2781] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:22.074041 18581 raft_consensus.cc:686] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 1 LEADER]: Becoming Leader. State: Replica: 3fbf8e22f8b04ff6ba02b815abe353b5, State: Running, Role: LEADER
I20260619 14:54:22.074586 18581 consensus_queue.cc:237] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866ac2bcc865460ca0b392cd91aad8f7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 39589 } } peers { permanent_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38879 } } peers { permanent_uuid: "7a07c89b928c469782b639d45fc01934" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41421 } }
I20260619 14:54:22.078845 18100 catalog_manager.cc:5495] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fbf8e22f8b04ff6ba02b815abe353b5 (127.0.241.1). New cstate: current_term: 1 leader_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "866ac2bcc865460ca0b392cd91aad8f7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 39589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fbf8e22f8b04ff6ba02b815abe353b5" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 38879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a07c89b928c469782b639d45fc01934" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41421 } health_report { overall_health: UNKNOWN } } }
W20260619 14:54:22.093003 18425 tablet.cc:2369] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:54:22.095021 18291 tablet.cc:2369] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:22.095903 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.090164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.0.241.1:38879)
0619 14:54:22.090364 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:22.090370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:22.090484 (+ 114us) server_negotiation.cc:407] Connection header received
0619 14:54:22.090710 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:22.090714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:22.090780 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:22.090917 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:22.091262 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.092038 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.093805 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.094237 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.094344 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:22.094780 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:22.095316 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:54:22.095481 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:22.095591 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 14:54:22.095656 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 14:54:22.097821 18558 tablet.cc:2369] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:22.103838 18379 raft_consensus.cc:1261] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934 [term 1 FOLLOWER]: Refusing update from remote peer 3fbf8e22f8b04ff6ba02b815abe353b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:22.103837 18512 raft_consensus.cc:1261] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7 [term 1 FOLLOWER]: Refusing update from remote peer 3fbf8e22f8b04ff6ba02b815abe353b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:22.104750 18586 consensus_queue.cc:1035] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "866ac2bcc865460ca0b392cd91aad8f7" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 39589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:22.105074 18581 consensus_queue.cc:1035] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a07c89b928c469782b639d45fc01934" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:22.118101 18589 mvcc.cc:205] Tried to move back new op lower bound from 7298584011167825920 to 7298584011061727232. Current Snapshot: MvccSnapshot[applied={T|T < 7298584011167825920}]
I20260619 14:54:22.119633 18591 mvcc.cc:205] Tried to move back new op lower bound from 7298584011167825920 to 7298584011061727232. Current Snapshot: MvccSnapshot[applied={T|T < 7298584011167825920}]
I20260619 14:54:22.130336 18100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:22.130614 18100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:22.134167 18100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b7db1b25e88492ca7b31f7aa1c9808c: Sending DeleteTablet for 3 replicas of tablet 4800200b2332400ea38fde5f569ee8d0
I20260619 14:54:22.135285 18225 tablet_service.cc:1514] Processing DeleteTablet for tablet 4800200b2332400ea38fde5f569ee8d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:22 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:58922
I20260619 14:54:22.135296 18492 tablet_service.cc:1514] Processing DeleteTablet for tablet 4800200b2332400ea38fde5f569ee8d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:22 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:47634
I20260619 14:54:22.135452 18359 tablet_service.cc:1514] Processing DeleteTablet for tablet 4800200b2332400ea38fde5f569ee8d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:22 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:48054
I20260619 14:54:22.137346 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 18162
I20260619 14:54:22.156332 18599 tablet_replica.cc:330] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5: stopping tablet replica
I20260619 14:54:22.156797 18599 raft_consensus.cc:2229] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:54:22.157455 18599 raft_consensus.cc:2258] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:22.158248 18600 tablet_replica.cc:330] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7: stopping tablet replica
I20260619 14:54:22.158646 18600 raft_consensus.cc:2229] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:22.159197 18600 raft_consensus.cc:2258] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:22.162940 18601 tablet_replica.cc:330] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934: stopping tablet replica
I20260619 14:54:22.163390 18601 raft_consensus.cc:2229] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:22.163923 18601 raft_consensus.cc:2258] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 14:54:22 dist-test-slave-p3ml krb5kdc[18056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:54:22.165036 18599 ts_tablet_manager.cc:1905] T 4800200b2332400ea38fde5f569ee8d0 P 3fbf8e22f8b04ff6ba02b815abe353b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:22.165180 18600 ts_tablet_manager.cc:1905] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:22.168273 18601 ts_tablet_manager.cc:1905] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:22.173158 18600 ts_tablet_manager.cc:1918] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:22.173349 18600 log.cc:1196] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/4800200b2332400ea38fde5f569ee8d0
I20260619 14:54:22.174036 18600 ts_tablet_manager.cc:1939] T 4800200b2332400ea38fde5f569ee8d0 P 866ac2bcc865460ca0b392cd91aad8f7: Deleting consensus metadata
I20260619 14:54:22.175650 18601 ts_tablet_manager.cc:1918] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:22.175823 18601 log.cc:1196] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/4800200b2332400ea38fde5f569ee8d0
I20260619 14:54:22.176036 18088 catalog_manager.cc:4841] TS 866ac2bcc865460ca0b392cd91aad8f7 (127.0.241.3:39589): tablet 4800200b2332400ea38fde5f569ee8d0 (table test-table [id=ac1a11762b8a4ff3ba408c375df06c06]) successfully deleted
I20260619 14:54:22.176385 18601 ts_tablet_manager.cc:1939] T 4800200b2332400ea38fde5f569ee8d0 P 7a07c89b928c469782b639d45fc01934: Deleting consensus metadata
I20260619 14:54:22.176530 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.139198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.0.241.1:38879)
0619 14:54:22.139414 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:22.139423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:22.149036 (+ 9613us) server_negotiation.cc:407] Connection header received
0619 14:54:22.149453 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:22.149459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:22.149536 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:22.149614 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:22.151431 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.153021 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.158175 (+ 5154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.159184 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.159313 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:22.166253 (+ 6940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:22.166268 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:22.166285 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:22.166348 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:22.170549 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:22.173071 (+ 2522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:22.173078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:22.173081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:22.173354 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:22.173982 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:22.173989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:22.173992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:22.174588 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:22.174858 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:22.175527 (+ 669us) server_negotiation.cc:299] Negotiation successful
0619 14:54:22.175921 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:54:22.177639 18285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:22.178278 18086 catalog_manager.cc:4841] TS 7a07c89b928c469782b639d45fc01934 (127.0.241.2:41421): tablet 4800200b2332400ea38fde5f569ee8d0 (table test-table [id=ac1a11762b8a4ff3ba408c375df06c06]) successfully deleted
I20260619 14:54:22.275929 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 18162
W20260619 14:54:22.294917 18085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.241.1:38879 (error 108)
I20260619 14:54:22.296135 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 18296
I20260619 14:54:22.296692 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.296191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:38879 (local address 127.0.0.1:58962)
0619 14:54:22.296371 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:22.296509 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 14:54:22.296943 18085 catalog_manager.cc:4584] TS 3fbf8e22f8b04ff6ba02b815abe353b5 (127.0.241.1:38879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4800200b2332400ea38fde5f569ee8d0: Network error: Client connection negotiation failed: client connection to 127.0.241.1:38879: connect: Connection refused (error 111)
Jun 19 14:54:22 dist-test-slave-p3ml krb5kdc[18056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:54:22.310711 18574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.296840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.0.241.2:41421)
0619 14:54:22.297008 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:22.297014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:22.297178 (+ 164us) server_negotiation.cc:407] Connection header received
0619 14:54:22.297385 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:22.297389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:22.297463 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:22.297566 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:22.298796 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.299601 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.300877 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.301259 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.301383 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:22.305395 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:22.305407 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:22.305422 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:22.305463 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:22.308063 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:22.308723 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:22.308730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:22.308732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:22.308848 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:22.309210 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:22.309217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:22.309220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:22.309579 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:22.309879 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:22.310172 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 14:54:22.310378 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:54:22.311317 18419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:22.350521 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.350058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:38879 (local address 127.0.0.1:58972)
0619 14:54:22.350254 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:22.350350 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 14:54:22.351613 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.351206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:38879 (local address 127.0.0.1:58982)
0619 14:54:22.351354 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:22.351472 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 14:54:22.400230 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 18296
I20260619 14:54:22.412360 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.411742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:38879 (local address 127.0.0.1:58992)
0619 14:54:22.412012 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:22.412129 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 14:54:22.413480 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.413045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:38879 (local address 127.0.0.1:59006)
0619 14:54:22.413223 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:22.413318 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 14:54:22.417723 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 18429
Jun 19 14:54:22 dist-test-slave-p3ml krb5kdc[18056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:54:22.432072 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.418427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.0.241.3:39589)
0619 14:54:22.418643 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:22.418651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:22.418672 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:54:22.418878 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:22.418890 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:22.418967 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:22.419084 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:22.420471 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.421191 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.422437 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:22.422838 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:22.422988 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:22.427137 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:22.427149 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:22.427164 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:22.427194 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:22.429707 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:22.430322 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:22.430328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:22.430331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:22.430448 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:22.430744 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:22.430747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:22.430749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:22.431058 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:22.431269 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:22.431643 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 14:54:22.431841 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:54:22.432716 18552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:22.494200 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.493584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:38879 (local address 127.0.0.1:59014)
0619 14:54:22.493896 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:22.494010 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260619 14:54:22.495345 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.494907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:38879 (local address 127.0.0.1:59022)
0619 14:54:22.495120 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:22.495220 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:38879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 14:54:22.522310 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 18429
I20260619 14:54:22.540709 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 18069
I20260619 14:54:22.542061 18131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:22.651165 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 18069
2026-06-19T14:54:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4217 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T14:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:22Z Disabled control of system clock
I20260619 14:54:22.724737 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45721
--webserver_interface=127.0.241.62
--webserver_port=0
--builtin_ntp_servers=127.0.241.20:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:45721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:54:23.046625 18611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:23.046906 18611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:23.046988 18611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:23.054177 18611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:23.054287 18611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:23.054333 18611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:23.054394 18611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:23.054437 18611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:23.063719 18611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:45721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:23.064930 18611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:23.066650 18611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:23.078305 18617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:23.078218 18616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:23.078254 18619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:23.078984 18611 server_base.cc:1034] running on GCE node
I20260619 14:54:23.079823 18611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:23.081085 18611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:23.082315 18611 hybrid_clock.cc:648] HybridClock initialized: now 1781880863082280 us; error 32 us; skew 500 ppm
I20260619 14:54:23.082687 18611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:54:23.084872 18611 webserver.cc:460] Webserver started at http://127.0.241.62:44809/ using document root <none> and password file <none>
I20260619 14:54:23.085547 18611 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:23.085675 18611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:23.086040 18611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:23.088874 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "89e2b24f647546498854e97a0e22178b"
format_stamp: "Formatted at 2026-06-19 14:54:23 on dist-test-slave-p3ml"
server_key: "b7ce851a0c9b94d7967d2d547a8fb244"
server_key_iv: "720f1a2a4773c075a2cb84756673c9bd"
server_key_version: "encryptionkey@0"
I20260619 14:54:23.089650 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "89e2b24f647546498854e97a0e22178b"
format_stamp: "Formatted at 2026-06-19 14:54:23 on dist-test-slave-p3ml"
server_key: "b7ce851a0c9b94d7967d2d547a8fb244"
server_key_iv: "720f1a2a4773c075a2cb84756673c9bd"
server_key_version: "encryptionkey@0"
I20260619 14:54:23.094596 18611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:54:23.097970 18625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:23.099239 18611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:54:23.099423 18611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "89e2b24f647546498854e97a0e22178b"
format_stamp: "Formatted at 2026-06-19 14:54:23 on dist-test-slave-p3ml"
server_key: "b7ce851a0c9b94d7967d2d547a8fb244"
server_key_iv: "720f1a2a4773c075a2cb84756673c9bd"
server_key_version: "encryptionkey@0"
I20260619 14:54:23.099689 18611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:23.145157 18611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:23.146016 18611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:23.146308 18611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:23.164930 18611 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:45721
I20260619 14:54:23.165019 18676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:45721 every 8 connection(s)
I20260619 14:54:23.166586 18611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:23.170217 18677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:23.174536 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 18611
I20260619 14:54:23.174708 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:23.175161 964 external_mini_cluster.cc:1428] Setting key 9de4af3026b1befdbc57077e50a5986e
I20260619 14:54:23.182086 18677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b: Bootstrap starting.
I20260619 14:54:23.185532 18677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:23.186791 18677 log.cc:824] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:23.190146 18677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b: No bootstrap required, opened a new log
I20260619 14:54:23.190224 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:23.176976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.0.241.62:45721)
0619 14:54:23.178034 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:23.178048 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:23.178098 (+ 50us) server_negotiation.cc:407] Connection header received
0619 14:54:23.180542 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:23.180566 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:23.181114 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:23.181501 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:23.182025 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:23.183969 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:23.185309 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:23.186359 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:23.186555 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:23.187729 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:23.187754 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:23.187775 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:23.188340 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:23.188432 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:23.188461 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:23.188566 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:23.188740 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 14:54:23.189059 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":333,"threads_started":1}
I20260619 14:54:23.197765 18677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e2b24f647546498854e97a0e22178b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45721 } }
I20260619 14:54:23.198076 18677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:23.198176 18677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e2b24f647546498854e97a0e22178b, State: Initialized, Role: FOLLOWER
I20260619 14:54:23.198707 18677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e2b24f647546498854e97a0e22178b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45721 } }
I20260619 14:54:23.198910 18677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:23.199005 18677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:23.199126 18677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:23.201480 18677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e2b24f647546498854e97a0e22178b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45721 } }
I20260619 14:54:23.201911 18677 leader_election.cc:302] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e2b24f647546498854e97a0e22178b; no voters:
I20260619 14:54:23.202461 18677 leader_election.cc:288] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:23.202603 18682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:23.203346 18682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [term 1 LEADER]: Becoming Leader. State: Replica: 89e2b24f647546498854e97a0e22178b, State: Running, Role: LEADER
I20260619 14:54:23.204197 18682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e2b24f647546498854e97a0e22178b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45721 } }
I20260619 14:54:23.204651 18677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:23.206799 18683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89e2b24f647546498854e97a0e22178b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e2b24f647546498854e97a0e22178b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45721 } } }
I20260619 14:54:23.207177 18683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:23.207715 18684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89e2b24f647546498854e97a0e22178b. Latest consensus state: current_term: 1 leader_uuid: "89e2b24f647546498854e97a0e22178b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e2b24f647546498854e97a0e22178b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45721 } } }
I20260619 14:54:23.208624 18684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:23.209892 18690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:23.214983 18690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:23.223836 18690 catalog_manager.cc:1347] Generated new cluster ID: 16e1924127a44ad484993d0a80933b21
I20260619 14:54:23.223980 18690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:23.234885 18690 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:23.236158 18690 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:23.241987 18690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b: Generated new TSK 0
I20260619 14:54:23.242779 18690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:23.253484 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45721
--builtin_ntp_servers=127.0.241.20:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:54:23.585807 18702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:23.586149 18702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:23.586305 18702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:23.593520 18702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:23.593611 18702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:23.593818 18702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:54:23.602757 18702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:23.604133 18702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:23.606021 18702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:23.619167 18707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:23.619122 18710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:23.619169 18708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:23.620189 18702 server_base.cc:1034] running on GCE node
I20260619 14:54:23.620750 18702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:23.621637 18702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:23.622900 18702 hybrid_clock.cc:648] HybridClock initialized: now 1781880863622828 us; error 59 us; skew 500 ppm
I20260619 14:54:23.623207 18702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:54:23.625439 18702 webserver.cc:460] Webserver started at http://127.0.241.1:34343/ using document root <none> and password file <none>
I20260619 14:54:23.626118 18702 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:23.626248 18702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:23.626561 18702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:23.629314 18702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "001f906388664717906ecf02d2b055a7"
format_stamp: "Formatted at 2026-06-19 14:54:23 on dist-test-slave-p3ml"
server_key: "4d4499869a1c1633f656b9cb441a9e7b"
server_key_iv: "2da93b93a445408ff81d345f8188372b"
server_key_version: "encryptionkey@0"
I20260619 14:54:23.630080 18702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "001f906388664717906ecf02d2b055a7"
format_stamp: "Formatted at 2026-06-19 14:54:23 on dist-test-slave-p3ml"
server_key: "4d4499869a1c1633f656b9cb441a9e7b"
server_key_iv: "2da93b93a445408ff81d345f8188372b"
server_key_version: "encryptionkey@0"
I20260619 14:54:23.634954 18702 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 14:54:23.638298 18716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:23.639586 18702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:54:23.639752 18702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "001f906388664717906ecf02d2b055a7"
format_stamp: "Formatted at 2026-06-19 14:54:23 on dist-test-slave-p3ml"
server_key: "4d4499869a1c1633f656b9cb441a9e7b"
server_key_iv: "2da93b93a445408ff81d345f8188372b"
server_key_version: "encryptionkey@0"
I20260619 14:54:23.639971 18702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:23.668335 18702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:23.669200 18702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:23.669489 18702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:23.670518 18702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:23.672008 18702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:23.672129 18702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:23.672269 18702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:23.672358 18702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:23.705811 18702 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:37071
I20260619 14:54:23.705917 18828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:37071 every 8 connection(s)
I20260619 14:54:23.707167 18702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:54:23.708200 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 18702
I20260619 14:54:23.708359 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:54:23.708940 964 external_mini_cluster.cc:1428] Setting key 676eb3acb0363c19dc7c93e16e30b451
I20260619 14:54:23.712080 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45721
--builtin_ntp_servers=127.0.241.20:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:54:23.722843 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:23.710937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:42471 (local address 127.0.241.62:45721)
0619 14:54:23.711450 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:23.711459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:23.715182 (+ 3723us) server_negotiation.cc:407] Connection header received
0619 14:54:23.715255 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:23.715262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:23.715349 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:23.715484 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:23.716717 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:23.717896 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:23.719393 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:23.720155 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:23.720267 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:23.721879 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:23.721888 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:23.721896 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:23.721922 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:23.721985 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:23.722002 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:23.722097 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:23.722292 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 14:54:23.722403 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":218,"threads_started":1}
I20260619 14:54:23.723768 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:23.710959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45721 (local address 127.0.241.1:42471)
0619 14:54:23.712115 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:23.712171 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:23.714829 (+ 2658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:23.715712 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:23.715720 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:23.715806 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:23.716524 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:23.716542 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:23.718048 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:23.718052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:23.719248 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:23.719260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:23.720418 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:23.720428 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:23.720735 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:23.721579 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:54:23.721582 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:23.721620 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:54:23.721691 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:23.722100 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:23.722108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:23.722114 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:54:23.722250 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 14:54:23.722531 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":379,"threads_started":1}
I20260619 14:54:23.725354 18829 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45721
I20260619 14:54:23.725669 18829 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:23.726367 18829 heartbeater.cc:511] Master 127.0.241.62:45721 requested a full tablet report, sending...
I20260619 14:54:23.728339 18642 ts_manager.cc:194] Registered new tserver with Master: 001f906388664717906ecf02d2b055a7 (127.0.241.1:37071)
I20260619 14:54:23.729786 18642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.241.1:42471
W20260619 14:54:24.040390 18833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:24.040652 18833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:24.040807 18833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:24.048350 18833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:24.048446 18833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:24.048605 18833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:54:24.057781 18833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:24.059201 18833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:24.061035 18833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:24.074000 18838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:24.074071 18839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:24.074049 18841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:24.074800 18833 server_base.cc:1034] running on GCE node
I20260619 14:54:24.075399 18833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:24.076333 18833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:24.077555 18833 hybrid_clock.cc:648] HybridClock initialized: now 1781880864077518 us; error 53 us; skew 500 ppm
I20260619 14:54:24.077863 18833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:54:24.079985 18833 webserver.cc:460] Webserver started at http://127.0.241.2:43977/ using document root <none> and password file <none>
I20260619 14:54:24.080631 18833 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:24.080756 18833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:24.081065 18833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:24.083835 18833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "fd7515358a184dc5b7c39549378a6617"
format_stamp: "Formatted at 2026-06-19 14:54:24 on dist-test-slave-p3ml"
server_key: "6a87de0622f452b000f56bb55902fa6c"
server_key_iv: "1fadc532904c5a6f47ddae93c0893a8a"
server_key_version: "encryptionkey@0"
I20260619 14:54:24.084594 18833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "fd7515358a184dc5b7c39549378a6617"
format_stamp: "Formatted at 2026-06-19 14:54:24 on dist-test-slave-p3ml"
server_key: "6a87de0622f452b000f56bb55902fa6c"
server_key_iv: "1fadc532904c5a6f47ddae93c0893a8a"
server_key_version: "encryptionkey@0"
I20260619 14:54:24.089344 18833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:54:24.092562 18847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:24.093765 18833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:54:24.093930 18833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "fd7515358a184dc5b7c39549378a6617"
format_stamp: "Formatted at 2026-06-19 14:54:24 on dist-test-slave-p3ml"
server_key: "6a87de0622f452b000f56bb55902fa6c"
server_key_iv: "1fadc532904c5a6f47ddae93c0893a8a"
server_key_version: "encryptionkey@0"
I20260619 14:54:24.094195 18833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:24.106190 18833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:24.106884 18833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:24.107141 18833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:24.108105 18833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:24.109508 18833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:24.109638 18833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:24.109757 18833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:24.109851 18833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:24.141227 18833 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:44345
I20260619 14:54:24.141309 18959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:44345 every 8 connection(s)
I20260619 14:54:24.142498 18833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:54:24.148416 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 18833
I20260619 14:54:24.148600 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:54:24.149027 964 external_mini_cluster.cc:1428] Setting key 40adf42c08de789a2adf419f7328d046
I20260619 14:54:24.152158 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45721
--builtin_ntp_servers=127.0.241.20:34231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:54:24.160755 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.145265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:57637 (local address 127.0.241.62:45721)
0619 14:54:24.145435 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:24.145439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:24.146666 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 14:54:24.148534 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:24.148539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:24.148626 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:24.148732 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:24.150020 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.151159 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.153663 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.154035 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.154138 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.159825 (+ 5687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:24.159833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:24.159838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:24.159865 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:24.159908 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:24.159925 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:24.160026 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:24.160254 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 14:54:24.160370 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:54:24.161562 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.145665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45721 (local address 127.0.241.2:57637)
0619 14:54:24.146504 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:24.146545 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:24.148263 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:24.149066 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:24.149073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:24.149215 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:24.149821 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.149846 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.151320 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.151326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.153505 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.153537 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.154254 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.154263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.154623 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.159412 (+ 4789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:54:24.159415 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:24.159509 (+ 94us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:54:24.159616 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:24.160037 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:24.160045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:24.160050 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:54:24.160188 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 14:54:24.160466 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":275,"threads_started":1}
I20260619 14:54:24.163007 18960 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45721
I20260619 14:54:24.163303 18960 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:24.164162 18960 heartbeater.cc:511] Master 127.0.241.62:45721 requested a full tablet report, sending...
I20260619 14:54:24.165707 18642 ts_manager.cc:194] Registered new tserver with Master: fd7515358a184dc5b7c39549378a6617 (127.0.241.2:44345)
I20260619 14:54:24.167061 18642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.241.2:57637
W20260619 14:54:24.488957 18963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:24.489290 18963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:24.489468 18963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:24.496816 18963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:24.496961 18963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:24.497176 18963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:54:24.506533 18963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:34231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:24.507822 18963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:24.509670 18963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:24.522646 18969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:24.522610 18968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:24.522702 18971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:24.523674 18963 server_base.cc:1034] running on GCE node
I20260619 14:54:24.524169 18963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:24.524948 18963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:24.526139 18963 hybrid_clock.cc:648] HybridClock initialized: now 1781880864526093 us; error 41 us; skew 500 ppm
I20260619 14:54:24.526430 18963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:54:24.528544 18963 webserver.cc:460] Webserver started at http://127.0.241.3:43455/ using document root <none> and password file <none>
I20260619 14:54:24.529182 18963 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:24.529292 18963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:24.529613 18963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:24.532284 18963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "45fd507168584ab18ba1a2bb2106ce88"
format_stamp: "Formatted at 2026-06-19 14:54:24 on dist-test-slave-p3ml"
server_key: "1b6e09fa47392c1a8d07a111f1f9d08d"
server_key_iv: "d8d230f299d57e3563084a71a4fad214"
server_key_version: "encryptionkey@0"
I20260619 14:54:24.533051 18963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "45fd507168584ab18ba1a2bb2106ce88"
format_stamp: "Formatted at 2026-06-19 14:54:24 on dist-test-slave-p3ml"
server_key: "1b6e09fa47392c1a8d07a111f1f9d08d"
server_key_iv: "d8d230f299d57e3563084a71a4fad214"
server_key_version: "encryptionkey@0"
I20260619 14:54:24.537710 18963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:54:24.540870 18977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:24.542058 18963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:54:24.542217 18963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "45fd507168584ab18ba1a2bb2106ce88"
format_stamp: "Formatted at 2026-06-19 14:54:24 on dist-test-slave-p3ml"
server_key: "1b6e09fa47392c1a8d07a111f1f9d08d"
server_key_iv: "d8d230f299d57e3563084a71a4fad214"
server_key_version: "encryptionkey@0"
I20260619 14:54:24.542423 18963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:24.553229 18963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:24.553944 18963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:24.554203 18963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:24.555125 18963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:24.556447 18963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:24.556565 18963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:24.556704 18963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:24.556792 18963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:24.588944 18963 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:33661
I20260619 14:54:24.588994 19089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:33661 every 8 connection(s)
I20260619 14:54:24.590216 18963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:54:24.598721 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 18963
I20260619 14:54:24.598894 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:54:24.599310 964 external_mini_cluster.cc:1428] Setting key 314423d06d130630a72d8b3bdbd3faa7
I20260619 14:54:24.602618 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.592694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:53793 (local address 127.0.241.62:45721)
0619 14:54:24.592911 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:24.592917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:24.594258 (+ 1341us) server_negotiation.cc:407] Connection header received
0619 14:54:24.596063 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:24.596067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:24.596134 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:24.596220 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:24.597301 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.598050 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.599810 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.600284 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.600421 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.601802 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:24.601808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:24.601812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:24.601832 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:24.601872 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:24.601888 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:24.601983 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:24.602214 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 14:54:24.602310 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 14:54:24.603663 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.593230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45721 (local address 127.0.241.3:53793)
0619 14:54:24.594090 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:24.594124 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:24.595832 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:24.596426 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:24.596434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:24.596542 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:24.597156 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.597174 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.598227 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.598230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.599655 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.599667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.600568 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.600583 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.600773 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.601485 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:54:24.601494 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:24.601535 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:54:24.601605 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:24.601994 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:24.602006 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:24.602016 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:54:24.602204 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 14:54:24.602585 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":276,"threads_started":1}
I20260619 14:54:24.604946 19090 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45721
I20260619 14:54:24.605226 19090 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:24.606004 19090 heartbeater.cc:511] Master 127.0.241.62:45721 requested a full tablet report, sending...
I20260619 14:54:24.607278 18642 ts_manager.cc:194] Registered new tserver with Master: 45fd507168584ab18ba1a2bb2106ce88 (127.0.241.3:33661)
I20260619 14:54:24.608099 18642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.241.3:53793
I20260619 14:54:24.614131 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:54:24.623227 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.618009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.0.241.62:45721)
0619 14:54:24.618189 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:24.618193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:24.618588 (+ 395us) server_negotiation.cc:407] Connection header received
0619 14:54:24.618713 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:24.618717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:24.618782 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:24.618861 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:24.619283 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.620002 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.621298 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.621677 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.621763 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.622693 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:24.622698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:24.622701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:24.622717 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:24.622747 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:24.622758 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:24.622826 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:24.622953 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 14:54:24.623021 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:54:24.626609 18642 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:54:24.633476 18642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:54:24.669338 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.645064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.0.241.1:37071)
0619 14:54:24.658785 (+ 13721us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:24.658798 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:24.658837 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:54:24.658964 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:24.658976 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:24.659311 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:24.659567 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:24.660941 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.663284 (+ 2343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.666350 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.667592 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.667721 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.667983 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:24.668745 (+ 762us) server_negotiation.cc:299] Negotiation successful
0619 14:54:24.668945 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13487,"thread_start_us":15564,"threads_started":1}
I20260619 14:54:24.670013 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.644890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:37071 (local address 127.0.0.1:46258)
0619 14:54:24.655447 (+ 10557us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:24.655493 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:24.655665 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:24.659929 (+ 4264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:24.659942 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:24.659987 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:24.660485 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.660502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.663487 (+ 2985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.663495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.666154 (+ 2659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.666167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.668104 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.668110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.668367 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.668442 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 14:54:24.668682 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 14:54:24.669704 (+ 1022us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10296,"mutex_wait_us":118,"thread_start_us":20171,"threads_started":1}
I20260619 14:54:24.672938 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.645076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:44345 (local address 127.0.0.1:50152)
0619 14:54:24.662493 (+ 17417us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:24.662512 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:24.662640 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:24.665905 (+ 3265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:24.665910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:24.665936 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:24.666421 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.666431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.668297 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.668302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.670537 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.670553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.672156 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.672162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.672338 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.672373 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:54:24.672471 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:54:24.672601 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17266,"mutex_wait_us":40,"thread_start_us":9937,"threads_started":1}
I20260619 14:54:24.672984 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.645835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.0.241.2:44345)
0619 14:54:24.665298 (+ 19463us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:24.665304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:24.665341 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:54:24.665441 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:24.665448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:24.665703 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:24.665927 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:24.666598 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.668147 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.670721 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.671970 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.672110 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.672357 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:24.672542 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 14:54:24.672689 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19290,"thread_start_us":19742,"threads_started":1}
I20260619 14:54:24.674388 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.650048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:33661 (local address 127.0.0.1:48350)
0619 14:54:24.665337 (+ 15289us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:24.665354 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:24.665467 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:24.666234 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:24.666238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:24.666287 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:24.666714 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.666725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.668612 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.668617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.671602 (+ 2985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.671614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.673616 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.673622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.673789 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.673822 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:54:24.673938 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 14:54:24.674074 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15159,"thread_start_us":12151,"threads_started":1}
I20260619 14:54:24.674243 18764 tablet_service.cc:1467] Processing CreateTablet for tablet af9399ba352d4fceab2ca186927b971f (DEFAULT_TABLE table=test-table [id=ceb16d860a9a4f918a89e33a289b22c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:24.675608 18764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9399ba352d4fceab2ca186927b971f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:24.676308 18895 tablet_service.cc:1467] Processing CreateTablet for tablet af9399ba352d4fceab2ca186927b971f (DEFAULT_TABLE table=test-table [id=ceb16d860a9a4f918a89e33a289b22c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:24.677444 18895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9399ba352d4fceab2ca186927b971f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:24.680126 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.650436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.0.241.3:33661)
0619 14:54:24.664672 (+ 14236us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:24.664680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:24.665464 (+ 784us) server_negotiation.cc:407] Connection header received
0619 14:54:24.665613 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:24.665620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:24.665791 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:24.665959 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:24.666902 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.668422 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.671791 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.673430 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.673583 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.673818 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:24.679595 (+ 5777us) server_negotiation.cc:299] Negotiation successful
0619 14:54:24.679761 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14046,"thread_start_us":14191,"threads_started":1}
I20260619 14:54:24.683665 19025 tablet_service.cc:1467] Processing CreateTablet for tablet af9399ba352d4fceab2ca186927b971f (DEFAULT_TABLE table=test-table [id=ceb16d860a9a4f918a89e33a289b22c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:24.685036 19025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af9399ba352d4fceab2ca186927b971f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:24.689720 19108 tablet_bootstrap.cc:492] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7: Bootstrap starting.
I20260619 14:54:24.692224 19109 tablet_bootstrap.cc:492] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617: Bootstrap starting.
I20260619 14:54:24.693885 19108 tablet_bootstrap.cc:654] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:24.694931 19108 log.cc:824] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:24.695724 19109 tablet_bootstrap.cc:654] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:24.696919 19109 log.cc:824] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:24.697992 19108 tablet_bootstrap.cc:492] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7: No bootstrap required, opened a new log
I20260619 14:54:24.698262 19108 ts_tablet_manager.cc:1397] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 14:54:24.701062 19109 tablet_bootstrap.cc:492] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617: No bootstrap required, opened a new log
I20260619 14:54:24.701376 19109 ts_tablet_manager.cc:1397] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 14:54:24.701972 19112 tablet_bootstrap.cc:492] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88: Bootstrap starting.
I20260619 14:54:24.705550 19112 tablet_bootstrap.cc:654] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:24.706748 19112 log.cc:824] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:24.708715 19108 raft_consensus.cc:348] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001f906388664717906ecf02d2b055a7" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37071 } } peers { permanent_uuid: "fd7515358a184dc5b7c39549378a6617" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 44345 } } peers { permanent_uuid: "45fd507168584ab18ba1a2bb2106ce88" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33661 } }
I20260619 14:54:24.709165 19108 raft_consensus.cc:374] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:24.709260 19108 raft_consensus.cc:729] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001f906388664717906ecf02d2b055a7, State: Initialized, Role: FOLLOWER
I20260619 14:54:24.709895 19108 consensus_queue.cc:260] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001f906388664717906ecf02d2b055a7" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37071 } } peers { permanent_uuid: "fd7515358a184dc5b7c39549378a6617" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 44345 } } peers { permanent_uuid: "45fd507168584ab18ba1a2bb2106ce88" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33661 } }
I20260619 14:54:24.710717 19109 raft_consensus.cc:348] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001f906388664717906ecf02d2b055a7" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37071 } } peers { permanent_uuid: "fd7515358a184dc5b7c39549378a6617" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 44345 } } peers { permanent_uuid: "45fd507168584ab18ba1a2bb2106ce88" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33661 } }
I20260619 14:54:24.711041 19109 raft_consensus.cc:374] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:24.711154 19109 raft_consensus.cc:729] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7515358a184dc5b7c39549378a6617, State: Initialized, Role: FOLLOWER
I20260619 14:54:24.711725 19109 consensus_queue.cc:260] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001f906388664717906ecf02d2b055a7" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37071 } } peers { permanent_uuid: "fd7515358a184dc5b7c39549378a6617" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 44345 } } peers { permanent_uuid: "45fd507168584ab18ba1a2bb2106ce88" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33661 } }
I20260619 14:54:24.715880 18960 heartbeater.cc:503] Master 127.0.241.62:45721 was elected leader, sending a full tablet report...
I20260619 14:54:24.716584 19109 ts_tablet_manager.cc:1428] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260619 14:54:24.719292 19112 tablet_bootstrap.cc:492] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88: No bootstrap required, opened a new log
I20260619 14:54:24.719668 19112 ts_tablet_manager.cc:1397] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.006s
I20260619 14:54:24.724120 18829 heartbeater.cc:503] Master 127.0.241.62:45721 was elected leader, sending a full tablet report...
I20260619 14:54:24.724785 19108 ts_tablet_manager.cc:1428] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7: Time spent starting tablet: real 0.026s user 0.008s sys 0.017s
I20260619 14:54:24.729302 19112 raft_consensus.cc:348] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001f906388664717906ecf02d2b055a7" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37071 } } peers { permanent_uuid: "fd7515358a184dc5b7c39549378a6617" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 44345 } } peers { permanent_uuid: "45fd507168584ab18ba1a2bb2106ce88" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33661 } }
I20260619 14:54:24.729651 19112 raft_consensus.cc:374] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:24.729768 19112 raft_consensus.cc:729] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fd507168584ab18ba1a2bb2106ce88, State: Initialized, Role: FOLLOWER
I20260619 14:54:24.730273 19112 consensus_queue.cc:260] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001f906388664717906ecf02d2b055a7" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37071 } } peers { permanent_uuid: "fd7515358a184dc5b7c39549378a6617" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 44345 } } peers { permanent_uuid: "45fd507168584ab18ba1a2bb2106ce88" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33661 } }
I20260619 14:54:24.731169 19090 heartbeater.cc:503] Master 127.0.241.62:45721 was elected leader, sending a full tablet report...
I20260619 14:54:24.731659 19112 ts_tablet_manager.cc:1428] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260619 14:54:24.842061 19091 tablet.cc:2369] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:54:24.894655 18961 tablet.cc:2369] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:24.916690 19116 raft_consensus.cc:482] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:24.916997 19116 raft_consensus.cc:504] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001f906388664717906ecf02d2b055a7" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37071 } } peers { permanent_uuid: "fd7515358a184dc5b7c39549378a6617" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 44345 } } peers { permanent_uuid: "45fd507168584ab18ba1a2bb2106ce88" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33661 } }
I20260619 14:54:24.918751 19116 leader_election.cc:288] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 001f906388664717906ecf02d2b055a7 (127.0.241.1:37071), fd7515358a184dc5b7c39549378a6617 (127.0.241.2:44345)
I20260619 14:54:24.925015 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.919354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:56539 (local address 127.0.241.2:44345)
0619 14:54:24.919588 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:24.919596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:24.919619 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:54:24.919712 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:24.919720 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:24.919826 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:24.919944 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:24.920431 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.921396 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.923210 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.924166 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.924257 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.924321 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:24.924661 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 14:54:24.924745 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 14:54:24.925068 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.919064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:44345 (local address 127.0.241.3:56539)
0619 14:54:24.919355 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:24.919377 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:24.919558 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:24.920013 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:24.920016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:24.920039 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:24.920274 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.920285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.921541 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.921546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.923059 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.923070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.924315 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.924322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.924506 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.924541 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:54:24.924634 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:54:24.924763 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260619 14:54:24.925771 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.919232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:50307 (local address 127.0.241.1:37071)
0619 14:54:24.919452 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:24.919461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:24.920133 (+ 672us) server_negotiation.cc:407] Connection header received
0619 14:54:24.920288 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:24.920294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:24.920395 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:24.920479 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:24.921094 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.921945 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.923811 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.924748 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.924865 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.924947 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:24.925293 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 14:54:24.925443 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:54:24.926251 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.919165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:37071 (local address 127.0.241.3:50307)
0619 14:54:24.920011 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:24.920028 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:24.920152 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:24.920522 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:24.920526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:24.920562 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:24.920925 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.920937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.922130 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.922133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.923632 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:24.923645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.924905 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.924912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:24.925091 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.925129 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:54:24.925950 (+ 821us) client_negotiation.cc:240] Negotiation successful
0619 14:54:24.926053 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":469,"threads_started":1}
I20260619 14:54:24.926192 18915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9399ba352d4fceab2ca186927b971f" candidate_uuid: "45fd507168584ab18ba1a2bb2106ce88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7515358a184dc5b7c39549378a6617" is_pre_election: true
I20260619 14:54:24.926806 18915 raft_consensus.cc:2454] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45fd507168584ab18ba1a2bb2106ce88 in term 0.
I20260619 14:54:24.927246 18784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9399ba352d4fceab2ca186927b971f" candidate_uuid: "45fd507168584ab18ba1a2bb2106ce88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001f906388664717906ecf02d2b055a7" is_pre_election: true
I20260619 14:54:24.927560 18981 leader_election.cc:302] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45fd507168584ab18ba1a2bb2106ce88, fd7515358a184dc5b7c39549378a6617; no voters:
I20260619 14:54:24.927834 18784 raft_consensus.cc:2454] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45fd507168584ab18ba1a2bb2106ce88 in term 0.
I20260619 14:54:24.927940 19116 raft_consensus.cc:2781] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:54:24.928108 19116 raft_consensus.cc:482] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:54:24.928211 19116 raft_consensus.cc:3037] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:24.930421 19116 raft_consensus.cc:504] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001f906388664717906ecf02d2b055a7" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37071 } } peers { permanent_uuid: "fd7515358a184dc5b7c39549378a6617" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 44345 } } peers { permanent_uuid: "45fd507168584ab18ba1a2bb2106ce88" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33661 } }
I20260619 14:54:24.930997 19116 leader_election.cc:288] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [CANDIDATE]: Term 1 election: Requested vote from peers 001f906388664717906ecf02d2b055a7 (127.0.241.1:37071), fd7515358a184dc5b7c39549378a6617 (127.0.241.2:44345)
I20260619 14:54:24.931314 18784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9399ba352d4fceab2ca186927b971f" candidate_uuid: "45fd507168584ab18ba1a2bb2106ce88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001f906388664717906ecf02d2b055a7"
I20260619 14:54:24.931552 18784 raft_consensus.cc:3037] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:24.931542 18915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af9399ba352d4fceab2ca186927b971f" candidate_uuid: "45fd507168584ab18ba1a2bb2106ce88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7515358a184dc5b7c39549378a6617"
I20260619 14:54:24.931733 18915 raft_consensus.cc:3037] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:24.933755 18784 raft_consensus.cc:2454] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45fd507168584ab18ba1a2bb2106ce88 in term 1.
I20260619 14:54:24.933841 18915 raft_consensus.cc:2454] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45fd507168584ab18ba1a2bb2106ce88 in term 1.
I20260619 14:54:24.934163 18979 leader_election.cc:302] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001f906388664717906ecf02d2b055a7, 45fd507168584ab18ba1a2bb2106ce88; no voters:
I20260619 14:54:24.934482 19116 raft_consensus.cc:2781] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:24.935024 19116 raft_consensus.cc:686] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 1 LEADER]: Becoming Leader. State: Replica: 45fd507168584ab18ba1a2bb2106ce88, State: Running, Role: LEADER
I20260619 14:54:24.935551 19116 consensus_queue.cc:237] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001f906388664717906ecf02d2b055a7" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37071 } } peers { permanent_uuid: "fd7515358a184dc5b7c39549378a6617" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 44345 } } peers { permanent_uuid: "45fd507168584ab18ba1a2bb2106ce88" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33661 } }
I20260619 14:54:24.940153 18642 catalog_manager.cc:5495] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45fd507168584ab18ba1a2bb2106ce88 (127.0.241.3). New cstate: current_term: 1 leader_uuid: "45fd507168584ab18ba1a2bb2106ce88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001f906388664717906ecf02d2b055a7" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd7515358a184dc5b7c39549378a6617" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 44345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45fd507168584ab18ba1a2bb2106ce88" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 33661 } health_report { overall_health: HEALTHY } } }
W20260619 14:54:24.962322 18830 tablet.cc:2369] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:24.966341 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.960869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48358 (local address 127.0.241.3:33661)
0619 14:54:24.961065 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:24.961072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:24.961174 (+ 102us) server_negotiation.cc:407] Connection header received
0619 14:54:24.961404 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:24.961408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:24.961481 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:24.961627 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:24.961992 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.962770 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.964300 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:24.964712 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:24.964855 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:24.965341 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:24.965788 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:54:24.965940 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:24.966027 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 14:54:24.966101 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:54:24.973796 18915 raft_consensus.cc:1261] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617 [term 1 FOLLOWER]: Refusing update from remote peer 45fd507168584ab18ba1a2bb2106ce88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:24.973798 18784 raft_consensus.cc:1261] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7 [term 1 FOLLOWER]: Refusing update from remote peer 45fd507168584ab18ba1a2bb2106ce88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:24.974713 19118 consensus_queue.cc:1035] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001f906388664717906ecf02d2b055a7" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 37071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:24.975121 19116 consensus_queue.cc:1035] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd7515358a184dc5b7c39549378a6617" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 44345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:24.998519 18642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:24.998795 18642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:25.002329 18642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89e2b24f647546498854e97a0e22178b: Sending DeleteTablet for 3 replicas of tablet af9399ba352d4fceab2ca186927b971f
I20260619 14:54:25.003402 18764 tablet_service.cc:1514] Processing DeleteTablet for tablet af9399ba352d4fceab2ca186927b971f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:24 UTC) from {username='slave'} at 127.0.0.1:46258
I20260619 14:54:25.003491 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet af9399ba352d4fceab2ca186927b971f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:24 UTC) from {username='slave'} at 127.0.0.1:48350
I20260619 14:54:25.004194 18895 tablet_service.cc:1514] Processing DeleteTablet for tablet af9399ba352d4fceab2ca186927b971f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:24 UTC) from {username='slave'} at 127.0.0.1:50152
I20260619 14:54:25.004995 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 18702
I20260619 14:54:25.022743 19133 tablet_replica.cc:330] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617: stopping tablet replica
I20260619 14:54:25.023332 19133 raft_consensus.cc:2229] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:25.024045 19133 raft_consensus.cc:2258] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:25.024706 19132 tablet_replica.cc:330] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88: stopping tablet replica
I20260619 14:54:25.025274 19132 raft_consensus.cc:2229] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:54:25.025990 19132 raft_consensus.cc:2258] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:25.028347 19132 ts_tablet_manager.cc:1905] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:25.031613 19133 ts_tablet_manager.cc:1905] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:25.035738 19132 ts_tablet_manager.cc:1918] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:25.035981 19132 log.cc:1196] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/af9399ba352d4fceab2ca186927b971f
I20260619 14:54:25.036602 19132 ts_tablet_manager.cc:1939] T af9399ba352d4fceab2ca186927b971f P 45fd507168584ab18ba1a2bb2106ce88: Deleting consensus metadata
I20260619 14:54:25.038501 19133 ts_tablet_manager.cc:1918] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:25.038467 18628 catalog_manager.cc:4841] TS 45fd507168584ab18ba1a2bb2106ce88 (127.0.241.3:33661): tablet af9399ba352d4fceab2ca186927b971f (table test-table [id=ceb16d860a9a4f918a89e33a289b22c9]) successfully deleted
I20260619 14:54:25.038664 19133 log.cc:1196] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/af9399ba352d4fceab2ca186927b971f
I20260619 14:54:25.039193 19133 ts_tablet_manager.cc:1939] T af9399ba352d4fceab2ca186927b971f P fd7515358a184dc5b7c39549378a6617: Deleting consensus metadata
I20260619 14:54:25.040603 19131 tablet_replica.cc:330] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7: stopping tablet replica
I20260619 14:54:25.041117 19131 raft_consensus.cc:2229] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:25.041412 18629 catalog_manager.cc:4841] TS fd7515358a184dc5b7c39549378a6617 (127.0.241.2:44345): tablet af9399ba352d4fceab2ca186927b971f (table test-table [id=ceb16d860a9a4f918a89e33a289b22c9]) successfully deleted
I20260619 14:54:25.041795 19131 raft_consensus.cc:2258] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:25.042482 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.006562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.0.241.1:37071)
0619 14:54:25.006781 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:25.006789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:25.021934 (+ 15145us) server_negotiation.cc:407] Connection header received
0619 14:54:25.024723 (+ 2789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:25.024733 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:25.024875 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:25.024970 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:25.025671 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:25.026865 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:25.029363 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:25.030061 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:25.030162 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:25.040596 (+ 10434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:25.040609 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:25.040625 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:25.041456 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:25.041518 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:25.041565 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:25.041662 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:25.041949 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:54:25.042076 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:54:25.043474 18824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:25.044065 19131 ts_tablet_manager.cc:1905] T af9399ba352d4fceab2ca186927b971f P 001f906388664717906ecf02d2b055a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:25.132622 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 18702
W20260619 14:54:25.150527 18627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.241.1:37071 (error 108)
I20260619 14:54:25.150704 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 18833
I20260619 14:54:25.152494 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.151861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:37071 (local address 127.0.0.1:46278)
0619 14:54:25.152163 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:25.152328 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260619 14:54:25.152701 18627 catalog_manager.cc:4584] TS 001f906388664717906ecf02d2b055a7 (127.0.241.1:37071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af9399ba352d4fceab2ca186927b971f: Network error: Client connection negotiation failed: client connection to 127.0.241.1:37071: connect: Connection refused (error 111)
I20260619 14:54:25.161613 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.151914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.0.241.2:44345)
0619 14:54:25.152106 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:25.152113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:25.152135 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:54:25.152368 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:25.152373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:25.152505 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:25.152629 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:25.153219 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:25.154031 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:25.155315 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:25.155821 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:25.155956 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:25.160088 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:25.160098 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:25.160111 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:25.160710 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:25.160820 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:25.160847 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:25.160959 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:25.161129 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 14:54:25.161234 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:54:25.162285 18955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:25.206278 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.205814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:37071 (local address 127.0.0.1:46288)
0619 14:54:25.205986 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:25.206099 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 14:54:25.207325 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.206990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:37071 (local address 127.0.0.1:46292)
0619 14:54:25.207144 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:25.207218 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 14:54:25.268136 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.267472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:37071 (local address 127.0.0.1:46294)
0619 14:54:25.267770 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:25.267890 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260619 14:54:25.269448 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.268995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:37071 (local address 127.0.0.1:46306)
0619 14:54:25.269206 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:25.269296 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 14:54:25.292562 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 18833
I20260619 14:54:25.310000 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 18963
I20260619 14:54:25.318073 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.310819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.0.241.3:33661)
0619 14:54:25.311024 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:25.311028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:25.311053 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:54:25.311337 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:25.311341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:25.311480 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:25.311589 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:25.312161 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:25.312985 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:25.314876 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:25.315289 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:25.315379 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:25.316619 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:25.316628 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:25.316641 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:25.317230 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:25.317273 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:25.317292 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:25.317370 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:25.317565 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 14:54:25.317642 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:54:25.318981 19085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:25.350334 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.349802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:37071 (local address 127.0.0.1:46322)
0619 14:54:25.350075 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:25.350170 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260619 14:54:25.351426 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.351041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:37071 (local address 127.0.0.1:46338)
0619 14:54:25.351230 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:25.351293 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 14:54:25.439726 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 18963
I20260619 14:54:25.455101 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 18611
I20260619 14:54:25.456246 18672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:25.561823 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.561180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:37071 (local address 127.0.0.1:46354)
0619 14:54:25.561524 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:25.561611 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:37071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260619 14:54:25.561795 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 18611
2026-06-19T14:54:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2911 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:25 dist-test-slave-p3ml krb5kdc[19142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:25 dist-test-slave-p3ml krb5kdc[19142](info): set up 2 sockets
Jun 19 14:54:25 dist-test-slave-p3ml krb5kdc[19142](info): commencing operation
krb5kdc: starting...
W20260619 14:54:26.667551 964 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:26 dist-test-slave-p3ml krb5kdc[19142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:54:26.894196 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:34891
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:34891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:27.215138 19155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:27.215404 19155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:27.215570 19155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:27.222625 19155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:27.222725 19155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:27.222772 19155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:27.222822 19155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:27.222860 19155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:27.232201 19155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:34891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:34891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:27.233377 19155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:27.234592 19155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:27.246681 19160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:27.246727 19163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:27.246635 19161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:27.247258 19155 server_base.cc:1034] running on GCE node
I20260619 14:54:27.248049 19155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:27.249409 19155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:27.250643 19155 hybrid_clock.cc:648] HybridClock initialized: now 1781880867250601 us; error 53 us; skew 500 ppm
I20260619 14:54:27.250976 19155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:27 dist-test-slave-p3ml krb5kdc[19142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:27.254104 19155 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:54:27.255730 19155 webserver.cc:460] Webserver started at http://127.0.241.62:32851/ using document root <none> and password file <none>
I20260619 14:54:27.256407 19155 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:27.256559 19155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:27.256870 19155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:27.259835 19155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "67f8e3a760424fd285487b8cc565c85b"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-p3ml"
server_key: "23300269befad03e48d398a79a753a96"
server_key_iv: "23ff1d8d160ed7cf2eab063061fe125f"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.260588 19155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "67f8e3a760424fd285487b8cc565c85b"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-p3ml"
server_key: "23300269befad03e48d398a79a753a96"
server_key_iv: "23ff1d8d160ed7cf2eab063061fe125f"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.265599 19155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:54:27.268978 19170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.270227 19155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:54:27.270411 19155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "67f8e3a760424fd285487b8cc565c85b"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-p3ml"
server_key: "23300269befad03e48d398a79a753a96"
server_key_iv: "23ff1d8d160ed7cf2eab063061fe125f"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.270601 19155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:27.305457 19155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:27.309926 19155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:27.310256 19155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:27.328776 19155 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:34891
I20260619 14:54:27.328809 19221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:34891 every 8 connection(s)
I20260619 14:54:27.330171 19155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:27.333734 19222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:27.336794 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 19155
I20260619 14:54:27.336977 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:27.337531 964 external_mini_cluster.cc:1428] Setting key 091a284394d0fa1462f9b28db05f10bc
I20260619 14:54:27.345156 19222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b: Bootstrap starting.
I20260619 14:54:27.348071 19222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:27.349006 19222 log.cc:824] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:54:27 dist-test-slave-p3ml krb5kdc[19142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:54:27.351948 19222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b: No bootstrap required, opened a new log
I20260619 14:54:27.359927 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.339320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.0.241.62:34891)
0619 14:54:27.340088 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.340101 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.340142 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:54:27.341925 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.341946 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.342329 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.342721 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:27.344241 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.345620 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.346934 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.347656 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.347849 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.352475 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:27.352483 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:27.352496 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:27.352532 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:27.355832 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.356628 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.356634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.356639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.356802 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.357220 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.357223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.357225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.357715 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:27.357941 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.358389 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.358758 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":341,"threads_started":1}
I20260619 14:54:27.360705 19222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f8e3a760424fd285487b8cc565c85b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34891 } }
I20260619 14:54:27.361035 19222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:27.361143 19222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f8e3a760424fd285487b8cc565c85b, State: Initialized, Role: FOLLOWER
I20260619 14:54:27.361768 19222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f8e3a760424fd285487b8cc565c85b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34891 } }
I20260619 14:54:27.362006 19222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:27.362211 19222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:27.362396 19222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:27.364827 19222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f8e3a760424fd285487b8cc565c85b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34891 } }
I20260619 14:54:27.365234 19222 leader_election.cc:302] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67f8e3a760424fd285487b8cc565c85b; no voters:
I20260619 14:54:27.365746 19222 leader_election.cc:288] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:27.365900 19227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:27.366765 19227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [term 1 LEADER]: Becoming Leader. State: Replica: 67f8e3a760424fd285487b8cc565c85b, State: Running, Role: LEADER
I20260619 14:54:27.367276 19227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f8e3a760424fd285487b8cc565c85b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34891 } }
I20260619 14:54:27.368088 19222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:27.370436 19229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67f8e3a760424fd285487b8cc565c85b. Latest consensus state: current_term: 1 leader_uuid: "67f8e3a760424fd285487b8cc565c85b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f8e3a760424fd285487b8cc565c85b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34891 } } }
I20260619 14:54:27.370755 19229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:27.371639 19235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:27.371498 19228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67f8e3a760424fd285487b8cc565c85b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f8e3a760424fd285487b8cc565c85b" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 34891 } } }
I20260619 14:54:27.371780 19228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:27.377956 19235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:27.386520 19235 catalog_manager.cc:1347] Generated new cluster ID: 24dee8ac19cf49bf9cfcb08e42f560c7
I20260619 14:54:27.386626 19235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:27.406106 19235 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:27.407864 19235 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:27.417171 19235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b: Generated new TSK 0
I20260619 14:54:27.418171 19235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:54:27.478912 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34891
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:27.799813 19248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:27.800066 19248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:27.800231 19248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:27.807283 19248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:27.807404 19248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:27.807610 19248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:54:27.816797 19248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:27.818010 19248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:27.819244 19248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:27.831610 19256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:27.831687 19253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:27.831676 19254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:27.832471 19248 server_base.cc:1034] running on GCE node
I20260619 14:54:27.832962 19248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:27.833734 19248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:27.834920 19248 hybrid_clock.cc:648] HybridClock initialized: now 1781880867834878 us; error 34 us; skew 500 ppm
I20260619 14:54:27.835177 19248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:27 dist-test-slave-p3ml krb5kdc[19142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:27.838195 19248 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:54:27.839617 19248 webserver.cc:460] Webserver started at http://127.0.241.1:41967/ using document root <none> and password file <none>
I20260619 14:54:27.840258 19248 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:27.840399 19248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:27.840710 19248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:27.843430 19248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "078eeebfe034471b9088314a0c660185"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-p3ml"
server_key: "34159d966810ac9650dfa356cd4cbee0"
server_key_iv: "00df8c845602f0f257664b1985395e58"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.844250 19248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "078eeebfe034471b9088314a0c660185"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-p3ml"
server_key: "34159d966810ac9650dfa356cd4cbee0"
server_key_iv: "00df8c845602f0f257664b1985395e58"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.848991 19248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:54:27.852197 19263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.853334 19248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:54:27.853497 19248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "078eeebfe034471b9088314a0c660185"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-p3ml"
server_key: "34159d966810ac9650dfa356cd4cbee0"
server_key_iv: "00df8c845602f0f257664b1985395e58"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.853689 19248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:27.878439 19248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:27.882246 19248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:27.882532 19248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:27.883467 19248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:27.884683 19248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:27.884784 19248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.884963 19248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:27.885042 19248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.917120 19248 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:34411
I20260619 14:54:27.917155 19375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:34411 every 8 connection(s)
I20260619 14:54:27.918427 19248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:54:27.924736 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 19248
I20260619 14:54:27.924902 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:54:27.925419 964 external_mini_cluster.cc:1428] Setting key 1e3fb7bc423a86bc7af5897ce76694ca
Jun 19 14:54:27 dist-test-slave-p3ml krb5kdc[19142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:54:27.940394 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.920796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:44773 (local address 127.0.241.62:34891)
0619 14:54:27.921234 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.921239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.921987 (+ 748us) server_negotiation.cc:407] Connection header received
0619 14:54:27.923906 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.923911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.923997 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.924116 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:27.926058 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.927094 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.929381 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.929788 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.929885 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.933875 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:27.933883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:27.933891 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:27.933933 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:27.937263 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.938184 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.938188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.938190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.938321 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.938787 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.938790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.938792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.939026 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:27.939140 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.939971 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.940141 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":258,"threads_started":1}
I20260619 14:54:27.941762 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.921196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34891 (local address 127.0.241.1:44773)
0619 14:54:27.921841 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:27.921877 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:27.923637 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:27.924374 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:27.924384 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:27.925128 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:27.925869 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.925892 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.927255 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.927259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.929246 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.929263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.930009 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.930018 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.930344 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.931050 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:27.931089 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:27.933664 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:27.937397 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:27.937407 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:27.937425 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:27.938071 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:27.938426 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:27.938433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:27.938440 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:27.938678 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:27.939150 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:27.939159 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:27.939590 (+ 431us) client_negotiation.cc:769] Sending connection context
0619 14:54:27.939875 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 14:54:27.940413 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":263,"threads_started":1}
I20260619 14:54:27.943300 19376 heartbeater.cc:347] Connected to a master server at 127.0.241.62:34891
I20260619 14:54:27.943661 19376 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:27.944339 19376 heartbeater.cc:511] Master 127.0.241.62:34891 requested a full tablet report, sending...
I20260619 14:54:27.946297 19187 ts_manager.cc:194] Registered new tserver with Master: 078eeebfe034471b9088314a0c660185 (127.0.241.1:34411)
I20260619 14:54:27.947796 19187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:44773
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:54:27.980885 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34891
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:28.294215 19382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:28.294453 19382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:28.294569 19382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:28.301746 19382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:28.301843 19382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:28.301954 19382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:54:28.311223 19382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:28.312425 19382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:28.313634 19382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:28.326164 19387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:28.326205 19388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:28.326117 19390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:28.327415 19382 server_base.cc:1034] running on GCE node
I20260619 14:54:28.327997 19382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:28.328908 19382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:28.330124 19382 hybrid_clock.cc:648] HybridClock initialized: now 1781880868330072 us; error 48 us; skew 500 ppm
I20260619 14:54:28.330415 19382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:28 dist-test-slave-p3ml krb5kdc[19142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:28.333563 19382 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:54:28.335088 19382 webserver.cc:460] Webserver started at http://127.0.241.2:36003/ using document root <none> and password file <none>
I20260619 14:54:28.335798 19382 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:28.335947 19382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:28.336236 19382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:28.338999 19382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "72e732a28ad0490e852c5834ed8a4dff"
format_stamp: "Formatted at 2026-06-19 14:54:28 on dist-test-slave-p3ml"
server_key: "9db1aa94a09d25c16d7f9be51c79cb37"
server_key_iv: "f9f4fb931c8d9d4fb7d06f402881f41c"
server_key_version: "encryptionkey@0"
I20260619 14:54:28.339841 19382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "72e732a28ad0490e852c5834ed8a4dff"
format_stamp: "Formatted at 2026-06-19 14:54:28 on dist-test-slave-p3ml"
server_key: "9db1aa94a09d25c16d7f9be51c79cb37"
server_key_iv: "f9f4fb931c8d9d4fb7d06f402881f41c"
server_key_version: "encryptionkey@0"
I20260619 14:54:28.344803 19382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 14:54:28.348384 19397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:28.349622 19382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:54:28.349792 19382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "72e732a28ad0490e852c5834ed8a4dff"
format_stamp: "Formatted at 2026-06-19 14:54:28 on dist-test-slave-p3ml"
server_key: "9db1aa94a09d25c16d7f9be51c79cb37"
server_key_iv: "f9f4fb931c8d9d4fb7d06f402881f41c"
server_key_version: "encryptionkey@0"
I20260619 14:54:28.349990 19382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:28.376976 19382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:28.380748 19382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:28.380991 19382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:28.381863 19382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:28.383026 19382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:28.383101 19382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:28.383175 19382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:28.383217 19382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:28.412962 19382 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:37625
I20260619 14:54:28.413009 19509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:37625 every 8 connection(s)
I20260619 14:54:28.414155 19382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:54:28.422060 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 19382
I20260619 14:54:28.422235 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:54:28.422690 964 external_mini_cluster.cc:1428] Setting key b79b80be8ab70feb4755b1cf3653e11d
Jun 19 14:54:28 dist-test-slave-p3ml krb5kdc[19142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:54:28.433941 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:28.416547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:55079 (local address 127.0.241.62:34891)
0619 14:54:28.416753 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:28.416760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:28.417950 (+ 1190us) server_negotiation.cc:407] Connection header received
0619 14:54:28.419753 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:28.419757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:28.419824 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:28.419917 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:28.421331 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:28.422268 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:28.423981 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:28.424440 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:28.424574 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:28.428206 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:28.428211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:28.428214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:28.428245 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:28.430849 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:28.431682 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:28.431685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:28.431688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:28.431813 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:28.432303 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:28.432306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:28.432308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:28.432550 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:28.432655 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:28.433368 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 14:54:28.433585 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:54:28.434877 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:28.417062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34891 (local address 127.0.241.2:55079)
0619 14:54:28.417788 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:28.417826 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:28.419507 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:28.420174 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:28.420182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:28.420661 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:28.421188 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:28.421204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:28.422413 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:28.422417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:28.423834 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:28.423850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:28.424648 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:28.424657 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:28.424901 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:28.425559 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:28.425580 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:28.427980 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:28.430993 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:28.431002 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:28.431016 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:28.431534 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:28.431926 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:28.431932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:28.431937 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:28.432190 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:28.432673 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:28.432679 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:28.433004 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 14:54:28.433263 (+ 259us) client_negotiation.cc:240] Negotiation successful
0619 14:54:28.433734 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":307,"threads_started":1}
I20260619 14:54:28.436271 19510 heartbeater.cc:347] Connected to a master server at 127.0.241.62:34891
I20260619 14:54:28.436573 19510 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:28.437237 19510 heartbeater.cc:511] Master 127.0.241.62:34891 requested a full tablet report, sending...
I20260619 14:54:28.438697 19187 ts_manager.cc:194] Registered new tserver with Master: 72e732a28ad0490e852c5834ed8a4dff (127.0.241.2:37625)
I20260619 14:54:28.439975 19187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:55079
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:54:28.476153 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34891
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:28.788360 19515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:28.788619 19515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:28.788710 19515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:28.795675 19515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:28.795768 19515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:28.795876 19515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:54:28.804888 19515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:34891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:28.806057 19515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:28.807206 19515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:28.819742 19523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:28.819847 19520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:28.819818 19521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:28.820919 19515 server_base.cc:1034] running on GCE node
I20260619 14:54:28.821445 19515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:28.822297 19515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:28.823526 19515 hybrid_clock.cc:648] HybridClock initialized: now 1781880868823483 us; error 52 us; skew 500 ppm
I20260619 14:54:28.823822 19515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:28 dist-test-slave-p3ml krb5kdc[19142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:28.826746 19515 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:54:28.828234 19515 webserver.cc:460] Webserver started at http://127.0.241.3:44769/ using document root <none> and password file <none>
I20260619 14:54:28.828889 19515 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:28.829028 19515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:28.829324 19515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:28.832124 19515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "5bf98f73c97b4c2ca4652332e33ce24e"
format_stamp: "Formatted at 2026-06-19 14:54:28 on dist-test-slave-p3ml"
server_key: "f30622746781875869d3701ec8f7a6c9"
server_key_iv: "fed6c8dafa7eed37da3ca09836db81fd"
server_key_version: "encryptionkey@0"
I20260619 14:54:28.832887 19515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "5bf98f73c97b4c2ca4652332e33ce24e"
format_stamp: "Formatted at 2026-06-19 14:54:28 on dist-test-slave-p3ml"
server_key: "f30622746781875869d3701ec8f7a6c9"
server_key_iv: "fed6c8dafa7eed37da3ca09836db81fd"
server_key_version: "encryptionkey@0"
I20260619 14:54:28.837850 19515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:54:28.841168 19530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:28.842389 19515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:54:28.842549 19515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "5bf98f73c97b4c2ca4652332e33ce24e"
format_stamp: "Formatted at 2026-06-19 14:54:28 on dist-test-slave-p3ml"
server_key: "f30622746781875869d3701ec8f7a6c9"
server_key_iv: "fed6c8dafa7eed37da3ca09836db81fd"
server_key_version: "encryptionkey@0"
I20260619 14:54:28.842768 19515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:28.874437 19515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:28.878762 19515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:28.879061 19515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:28.880044 19515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:28.881322 19515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:28.881425 19515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:28.881573 19515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:28.881660 19515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:28.916759 19515 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:36179
I20260619 14:54:28.916796 19642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:36179 every 8 connection(s)
I20260619 14:54:28.918401 19515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:54:28.927587 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 19515
I20260619 14:54:28.927774 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:54:28.928197 964 external_mini_cluster.cc:1428] Setting key d92c085e4dabad7243f95a34e2dd8ce3
Jun 19 14:54:28 dist-test-slave-p3ml krb5kdc[19142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:54:28.938508 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:28.921224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:55097 (local address 127.0.241.62:34891)
0619 14:54:28.921424 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:28.921428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:28.922595 (+ 1167us) server_negotiation.cc:407] Connection header received
0619 14:54:28.924463 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:28.924469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:28.924585 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:28.924700 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:28.926158 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:28.926928 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:28.928706 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:28.929357 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:28.929476 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:28.933110 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:28.933116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:28.933121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:28.933166 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:28.935576 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:28.936271 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:28.936277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:28.936280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:28.936415 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:28.936852 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:28.936857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:28.936860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:28.937127 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:28.937292 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:28.937944 (+ 652us) server_negotiation.cc:299] Negotiation successful
0619 14:54:28.938169 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:54:28.939389 19645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:28.921672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:34891 (local address 127.0.241.3:55097)
0619 14:54:28.922437 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:28.922473 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:28.924198 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:28.924918 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:28.924926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:28.925393 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:28.925954 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:28.925970 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:28.927141 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:28.927146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:28.928534 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:28.928546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:28.929579 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:28.929586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:28.929890 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:28.930662 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:28.930696 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:28.932915 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:28.935724 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:28.935732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:28.935745 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:28.936142 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:28.936534 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:28.936539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:28.936543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:28.936731 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:28.937265 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:28.937272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:28.937578 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 14:54:28.937836 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 14:54:28.938308 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":327,"threads_started":1}
I20260619 14:54:28.940768 19643 heartbeater.cc:347] Connected to a master server at 127.0.241.62:34891
I20260619 14:54:28.941071 19643 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:28.941882 19643 heartbeater.cc:511] Master 127.0.241.62:34891 requested a full tablet report, sending...
I20260619 14:54:28.943325 19187 ts_manager.cc:194] Registered new tserver with Master: 5bf98f73c97b4c2ca4652332e33ce24e (127.0.241.3:36179)
I20260619 14:54:28.944124 19187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:55097
I20260619 14:54:28.950428 19376 heartbeater.cc:503] Master 127.0.241.62:34891 was elected leader, sending a full tablet report...
I20260619 14:54:28.954954 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:54:28.972390 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:28.959051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.0.241.62:34891)
0619 14:54:28.959206 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:28.959210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:28.959723 (+ 513us) server_negotiation.cc:407] Connection header received
0619 14:54:28.959848 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:28.959851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:28.959906 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:28.959979 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:28.961233 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:28.961892 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:28.963809 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:28.964399 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:28.964467 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:28.967145 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:28.967150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:28.967153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:28.967181 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:28.970051 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:28.970696 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:28.970700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:28.970702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:28.970830 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:28.971192 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:28.971195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:28.971196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:28.971537 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:28.971740 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:28.972014 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 14:54:28.972166 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:54:28.976081 19187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:54:28.983764 19187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:54:29.006034 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:28.995587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:36179 (local address 127.0.0.1:45598)
0619 14:54:28.997564 (+ 1977us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:28.997586 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:28.997693 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:28.998773 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:28.998779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:28.998804 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:28.999220 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:28.999236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.001362 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.001367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:29.003522 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:29.003535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.005372 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.005380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:29.005547 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.005574 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:54:29.005637 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:54:29.005723 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1840,"thread_start_us":1197,"threads_started":1}
I20260619 14:54:29.006588 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:28.996149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.0.241.3:36179)
0619 14:54:28.998227 (+ 2078us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:28.998234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:28.998265 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:54:28.998353 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:28.998360 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:28.998562 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:28.999451 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:28.999699 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.001202 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.003751 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.005143 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.005848 (+ 705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.006042 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:29.006120 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 14:54:29.006281 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1894,"thread_start_us":2040,"threads_started":1}
I20260619 14:54:29.007297 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:28.995800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.0.241.1:34411)
0619 14:54:28.997357 (+ 1557us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:28.997365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:28.997391 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:54:28.997486 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:28.997494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:28.997688 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:28.997907 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:28.998588 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.000257 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.003004 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.004500 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.004632 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.004856 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:29.006931 (+ 2075us) server_negotiation.cc:299] Negotiation successful
0619 14:54:29.007073 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1404,"thread_start_us":1231,"threads_started":1}
I20260619 14:54:29.007794 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:28.995493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:34411 (local address 127.0.0.1:49620)
0619 14:54:28.996250 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:28.996300 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:28.996487 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:28.997900 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:28.997907 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:28.997934 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:28.998401 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:28.998415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.000412 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.000418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:29.002829 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:29.002847 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.004994 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.005047 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:29.005290 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.005359 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 14:54:29.005481 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 14:54:29.005669 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":425,"threads_started":1}
I20260619 14:54:29.012091 19578 tablet_service.cc:1467] Processing CreateTablet for tablet 362d1b3d66794950885b3c6de5f431e0 (DEFAULT_TABLE table=test-table [id=0b88847efe604089bcec228ac10d8b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:29.011976 19311 tablet_service.cc:1467] Processing CreateTablet for tablet 362d1b3d66794950885b3c6de5f431e0 (DEFAULT_TABLE table=test-table [id=0b88847efe604089bcec228ac10d8b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:29.013284 19578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362d1b3d66794950885b3c6de5f431e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:29.013402 19311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362d1b3d66794950885b3c6de5f431e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:29.017541 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:28.998403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:37625 (local address 127.0.0.1:41078)
0619 14:54:29.006535 (+ 8132us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:29.006551 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:29.006651 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:29.010260 (+ 3609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:29.010264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:29.010288 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:29.010448 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:29.010458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.012589 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.012594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:29.015017 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:29.015032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.016809 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.016815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:29.017000 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.017032 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:54:29.017144 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:54:29.017283 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8057,"thread_start_us":10773,"threads_started":1}
I20260619 14:54:29.017675 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:28.998917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.0.241.2:37625)
0619 14:54:29.009683 (+ 10766us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:29.009692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:29.009718 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:54:29.009808 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:29.009817 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:29.010047 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:29.010611 (+ 564us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:29.010862 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.012436 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.015203 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.016641 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.016768 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.016933 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:29.017192 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 14:54:29.017352 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10601,"thread_start_us":10489,"threads_started":1}
I20260619 14:54:29.021190 19445 tablet_service.cc:1467] Processing CreateTablet for tablet 362d1b3d66794950885b3c6de5f431e0 (DEFAULT_TABLE table=test-table [id=0b88847efe604089bcec228ac10d8b73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:29.022496 19445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362d1b3d66794950885b3c6de5f431e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:29.025569 19661 tablet_bootstrap.cc:492] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e: Bootstrap starting.
I20260619 14:54:29.029505 19661 tablet_bootstrap.cc:654] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:29.030496 19661 log.cc:824] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:29.033071 19661 tablet_bootstrap.cc:492] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e: No bootstrap required, opened a new log
I20260619 14:54:29.033324 19661 ts_tablet_manager.cc:1397] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 14:54:29.035971 19664 tablet_bootstrap.cc:492] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff: Bootstrap starting.
I20260619 14:54:29.036263 19662 tablet_bootstrap.cc:492] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185: Bootstrap starting.
I20260619 14:54:29.039418 19664 tablet_bootstrap.cc:654] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:29.039417 19662 tablet_bootstrap.cc:654] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:29.040632 19662 log.cc:824] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:29.040644 19664 log.cc:824] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:29.040707 19661 raft_consensus.cc:348] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078eeebfe034471b9088314a0c660185" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 34411 } } peers { permanent_uuid: "5bf98f73c97b4c2ca4652332e33ce24e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 36179 } } peers { permanent_uuid: "72e732a28ad0490e852c5834ed8a4dff" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37625 } }
I20260619 14:54:29.041108 19661 raft_consensus.cc:374] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:29.041271 19661 raft_consensus.cc:729] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf98f73c97b4c2ca4652332e33ce24e, State: Initialized, Role: FOLLOWER
I20260619 14:54:29.042068 19661 consensus_queue.cc:260] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078eeebfe034471b9088314a0c660185" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 34411 } } peers { permanent_uuid: "5bf98f73c97b4c2ca4652332e33ce24e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 36179 } } peers { permanent_uuid: "72e732a28ad0490e852c5834ed8a4dff" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37625 } }
I20260619 14:54:29.043222 19643 heartbeater.cc:503] Master 127.0.241.62:34891 was elected leader, sending a full tablet report...
I20260619 14:54:29.043763 19664 tablet_bootstrap.cc:492] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff: No bootstrap required, opened a new log
I20260619 14:54:29.043891 19661 ts_tablet_manager.cc:1428] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260619 14:54:29.043994 19664 ts_tablet_manager.cc:1397] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260619 14:54:29.044245 19662 tablet_bootstrap.cc:492] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185: No bootstrap required, opened a new log
I20260619 14:54:29.044538 19662 ts_tablet_manager.cc:1397] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20260619 14:54:29.052204 19664 raft_consensus.cc:348] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078eeebfe034471b9088314a0c660185" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 34411 } } peers { permanent_uuid: "5bf98f73c97b4c2ca4652332e33ce24e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 36179 } } peers { permanent_uuid: "72e732a28ad0490e852c5834ed8a4dff" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37625 } }
I20260619 14:54:29.052529 19664 raft_consensus.cc:374] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:29.052675 19664 raft_consensus.cc:729] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e732a28ad0490e852c5834ed8a4dff, State: Initialized, Role: FOLLOWER
I20260619 14:54:29.052994 19662 raft_consensus.cc:348] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078eeebfe034471b9088314a0c660185" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 34411 } } peers { permanent_uuid: "5bf98f73c97b4c2ca4652332e33ce24e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 36179 } } peers { permanent_uuid: "72e732a28ad0490e852c5834ed8a4dff" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37625 } }
I20260619 14:54:29.053365 19662 raft_consensus.cc:374] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:29.053297 19664 consensus_queue.cc:260] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078eeebfe034471b9088314a0c660185" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 34411 } } peers { permanent_uuid: "5bf98f73c97b4c2ca4652332e33ce24e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 36179 } } peers { permanent_uuid: "72e732a28ad0490e852c5834ed8a4dff" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37625 } }
I20260619 14:54:29.053493 19662 raft_consensus.cc:729] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078eeebfe034471b9088314a0c660185, State: Initialized, Role: FOLLOWER
I20260619 14:54:29.054067 19662 consensus_queue.cc:260] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078eeebfe034471b9088314a0c660185" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 34411 } } peers { permanent_uuid: "5bf98f73c97b4c2ca4652332e33ce24e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 36179 } } peers { permanent_uuid: "72e732a28ad0490e852c5834ed8a4dff" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37625 } }
I20260619 14:54:29.054420 19510 heartbeater.cc:503] Master 127.0.241.62:34891 was elected leader, sending a full tablet report...
I20260619 14:54:29.055123 19664 ts_tablet_manager.cc:1428] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260619 14:54:29.055894 19662 ts_tablet_manager.cc:1428] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
W20260619 14:54:29.166119 19511 tablet.cc:2369] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:54:29.170032 19644 tablet.cc:2369] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:54:29.170701 19377 tablet.cc:2369] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:29.438210 19669 raft_consensus.cc:482] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:29.438514 19669 raft_consensus.cc:504] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078eeebfe034471b9088314a0c660185" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 34411 } } peers { permanent_uuid: "5bf98f73c97b4c2ca4652332e33ce24e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 36179 } } peers { permanent_uuid: "72e732a28ad0490e852c5834ed8a4dff" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37625 } }
I20260619 14:54:29.440234 19669 leader_election.cc:288] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bf98f73c97b4c2ca4652332e33ce24e (127.0.241.3:36179), 72e732a28ad0490e852c5834ed8a4dff (127.0.241.2:37625)
I20260619 14:54:29.447427 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:29.440589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:38493 (local address 127.0.241.3:36179)
0619 14:54:29.440760 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:29.440765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:29.441343 (+ 578us) server_negotiation.cc:407] Connection header received
0619 14:54:29.441656 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:29.441665 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:29.441743 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:29.441797 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:29.442559 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.443410 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.445371 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.446372 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.446471 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.446577 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:29.447034 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 14:54:29.447177 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:54:29.447530 19670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:29.440582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:36179 (local address 127.0.241.1:38493)
0619 14:54:29.441180 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:29.441222 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:29.441393 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:29.442001 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:29.442005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:29.442036 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:29.442396 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:29.442410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.443591 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.443596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:29.445216 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:29.445229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.446580 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.446586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:29.446776 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.446812 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:54:29.446985 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 14:54:29.447138 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":1,"thread_start_us":318,"threads_started":1}
I20260619 14:54:29.447913 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:29.441446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:51615 (local address 127.0.241.2:37625)
0619 14:54:29.441644 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:29.441650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:29.442010 (+ 360us) server_negotiation.cc:407] Connection header received
0619 14:54:29.442156 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:29.442164 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:29.442240 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:29.442346 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:29.442951 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.443797 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.445862 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.446842 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.446980 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.447068 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:29.447519 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 14:54:29.447659 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:54:29.447952 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:29.441262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:37625 (local address 127.0.241.1:51615)
0619 14:54:29.441901 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:29.441919 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:29.442021 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:29.442366 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:29.442372 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:29.442389 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:29.442785 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:29.442797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.443927 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.443931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:29.445668 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:29.445681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.447146 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.447152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:29.447321 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.447356 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:54:29.447481 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 14:54:29.447647 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":333,"threads_started":1}
I20260619 14:54:29.448716 19598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362d1b3d66794950885b3c6de5f431e0" candidate_uuid: "078eeebfe034471b9088314a0c660185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf98f73c97b4c2ca4652332e33ce24e" is_pre_election: true
I20260619 14:54:29.449013 19465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362d1b3d66794950885b3c6de5f431e0" candidate_uuid: "078eeebfe034471b9088314a0c660185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e732a28ad0490e852c5834ed8a4dff" is_pre_election: true
I20260619 14:54:29.449226 19598 raft_consensus.cc:2454] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078eeebfe034471b9088314a0c660185 in term 0.
I20260619 14:54:29.449524 19465 raft_consensus.cc:2454] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 078eeebfe034471b9088314a0c660185 in term 0.
I20260619 14:54:29.449896 19267 leader_election.cc:302] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 078eeebfe034471b9088314a0c660185, 5bf98f73c97b4c2ca4652332e33ce24e; no voters:
I20260619 14:54:29.450304 19669 raft_consensus.cc:2781] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:54:29.450464 19669 raft_consensus.cc:482] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:54:29.450587 19669 raft_consensus.cc:3037] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:29.452785 19669 raft_consensus.cc:504] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078eeebfe034471b9088314a0c660185" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 34411 } } peers { permanent_uuid: "5bf98f73c97b4c2ca4652332e33ce24e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 36179 } } peers { permanent_uuid: "72e732a28ad0490e852c5834ed8a4dff" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37625 } }
I20260619 14:54:29.453488 19669 leader_election.cc:288] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [CANDIDATE]: Term 1 election: Requested vote from peers 5bf98f73c97b4c2ca4652332e33ce24e (127.0.241.3:36179), 72e732a28ad0490e852c5834ed8a4dff (127.0.241.2:37625)
I20260619 14:54:29.453848 19598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362d1b3d66794950885b3c6de5f431e0" candidate_uuid: "078eeebfe034471b9088314a0c660185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bf98f73c97b4c2ca4652332e33ce24e"
I20260619 14:54:29.453927 19465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362d1b3d66794950885b3c6de5f431e0" candidate_uuid: "078eeebfe034471b9088314a0c660185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e732a28ad0490e852c5834ed8a4dff"
I20260619 14:54:29.454046 19598 raft_consensus.cc:3037] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:29.454154 19465 raft_consensus.cc:3037] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:29.456374 19598 raft_consensus.cc:2454] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078eeebfe034471b9088314a0c660185 in term 1.
I20260619 14:54:29.456614 19465 raft_consensus.cc:2454] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 078eeebfe034471b9088314a0c660185 in term 1.
I20260619 14:54:29.456761 19267 leader_election.cc:302] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 078eeebfe034471b9088314a0c660185, 5bf98f73c97b4c2ca4652332e33ce24e; no voters:
I20260619 14:54:29.457052 19669 raft_consensus.cc:2781] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:29.457499 19669 raft_consensus.cc:686] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 1 LEADER]: Becoming Leader. State: Replica: 078eeebfe034471b9088314a0c660185, State: Running, Role: LEADER
I20260619 14:54:29.458039 19669 consensus_queue.cc:237] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078eeebfe034471b9088314a0c660185" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 34411 } } peers { permanent_uuid: "5bf98f73c97b4c2ca4652332e33ce24e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 36179 } } peers { permanent_uuid: "72e732a28ad0490e852c5834ed8a4dff" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37625 } }
I20260619 14:54:29.462430 19187 catalog_manager.cc:5495] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 reported cstate change: term changed from 0 to 1, leader changed from <none> to 078eeebfe034471b9088314a0c660185 (127.0.241.1). New cstate: current_term: 1 leader_uuid: "078eeebfe034471b9088314a0c660185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078eeebfe034471b9088314a0c660185" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 34411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bf98f73c97b4c2ca4652332e33ce24e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 36179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e732a28ad0490e852c5834ed8a4dff" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37625 } health_report { overall_health: UNKNOWN } } }
I20260619 14:54:29.574287 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:29.565581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49624 (local address 127.0.241.1:34411)
0619 14:54:29.566474 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:29.566479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:29.566513 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:54:29.566638 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:29.566642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:29.566773 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:29.566978 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:29.567539 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.568810 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.571298 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.571966 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.572114 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.572852 (+ 738us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:29.573554 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:54:29.573797 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:29.573923 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 14:54:29.573987 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":642,"threads_started":1}
I20260619 14:54:29.581961 19598 raft_consensus.cc:1261] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e [term 1 FOLLOWER]: Refusing update from remote peer 078eeebfe034471b9088314a0c660185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:29.581984 19465 raft_consensus.cc:1261] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff [term 1 FOLLOWER]: Refusing update from remote peer 078eeebfe034471b9088314a0c660185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:29.582877 19672 consensus_queue.cc:1035] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bf98f73c97b4c2ca4652332e33ce24e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:29.583211 19669 consensus_queue.cc:1035] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e732a28ad0490e852c5834ed8a4dff" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:29.592065 19677 mvcc.cc:205] Tried to move back new op lower bound from 7298584041797894144 to 7298584041303957504. Current Snapshot: MvccSnapshot[applied={T|T < 7298584041797894144}]
I20260619 14:54:29.605201 19187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:29.605473 19187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:29.605929 19681 mvcc.cc:205] Tried to move back new op lower bound from 7298584041797894144 to 7298584041303957504. Current Snapshot: MvccSnapshot[applied={T|T < 7298584041797894144}]
I20260619 14:54:29.610436 19187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67f8e3a760424fd285487b8cc565c85b: Sending DeleteTablet for 3 replicas of tablet 362d1b3d66794950885b3c6de5f431e0
I20260619 14:54:29.611481 19445 tablet_service.cc:1514] Processing DeleteTablet for tablet 362d1b3d66794950885b3c6de5f431e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:29 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:41078
I20260619 14:54:29.611938 19311 tablet_service.cc:1514] Processing DeleteTablet for tablet 362d1b3d66794950885b3c6de5f431e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:29 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:49620
I20260619 14:54:29.612301 19578 tablet_service.cc:1514] Processing DeleteTablet for tablet 362d1b3d66794950885b3c6de5f431e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:29 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:45598
I20260619 14:54:29.612922 19690 tablet_replica.cc:330] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185: stopping tablet replica
I20260619 14:54:29.613106 19689 tablet_replica.cc:330] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff: stopping tablet replica
I20260619 14:54:29.613106 19445 tablet_service.cc:1514] Processing DeleteTablet for tablet 362d1b3d66794950885b3c6de5f431e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:29 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:41078
I20260619 14:54:29.613459 19690 raft_consensus.cc:2229] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:54:29.613539 19689 raft_consensus.cc:2229] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:29.613982 19689 raft_consensus.cc:2258] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:29.614218 19690 raft_consensus.cc:2258] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:29.615103 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 19248
I20260619 14:54:29.615782 19689 ts_tablet_manager.cc:1905] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:29.616073 19690 ts_tablet_manager.cc:1905] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:29.623582 19690 ts_tablet_manager.cc:1918] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:29.623585 19689 ts_tablet_manager.cc:1918] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:29.623747 19689 log.cc:1196] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/362d1b3d66794950885b3c6de5f431e0
I20260619 14:54:29.623884 19690 log.cc:1196] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/362d1b3d66794950885b3c6de5f431e0
I20260619 14:54:29.624352 19689 ts_tablet_manager.cc:1939] T 362d1b3d66794950885b3c6de5f431e0 P 72e732a28ad0490e852c5834ed8a4dff: Deleting consensus metadata
I20260619 14:54:29.624548 19690 ts_tablet_manager.cc:1939] T 362d1b3d66794950885b3c6de5f431e0 P 078eeebfe034471b9088314a0c660185: Deleting consensus metadata
I20260619 14:54:29.626329 19174 catalog_manager.cc:4841] TS 72e732a28ad0490e852c5834ed8a4dff (127.0.241.2:37625): tablet 362d1b3d66794950885b3c6de5f431e0 (table test-table [id=0b88847efe604089bcec228ac10d8b73]) successfully deleted
I20260619 14:54:29.626363 19172 catalog_manager.cc:4841] TS 078eeebfe034471b9088314a0c660185 (127.0.241.1:34411): tablet 362d1b3d66794950885b3c6de5f431e0 (table test-table [id=0b88847efe604089bcec228ac10d8b73]) successfully deleted
W20260619 14:54:29.626856 19174 catalog_manager.cc:4808] TS 72e732a28ad0490e852c5834ed8a4dff (127.0.241.2:37625): delete failed for tablet 362d1b3d66794950885b3c6de5f431e0 because the tablet was not found. No further retry: Not found: Tablet not found: 362d1b3d66794950885b3c6de5f431e0
I20260619 14:54:29.635246 19691 tablet_replica.cc:330] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e: stopping tablet replica
I20260619 14:54:29.635797 19691 raft_consensus.cc:2229] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:29.636332 19691 raft_consensus.cc:2258] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:29.638059 19691 ts_tablet_manager.cc:1905] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:29.644738 19691 ts_tablet_manager.cc:1918] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:29.645331 19691 log.cc:1196] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/362d1b3d66794950885b3c6de5f431e0
Jun 19 14:54:29 dist-test-slave-p3ml krb5kdc[19142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:54:29.645859 19691 ts_tablet_manager.cc:1939] T 362d1b3d66794950885b3c6de5f431e0 P 5bf98f73c97b4c2ca4652332e33ce24e: Deleting consensus metadata
I20260619 14:54:29.647657 19174 catalog_manager.cc:4841] TS 5bf98f73c97b4c2ca4652332e33ce24e (127.0.241.3:36179): tablet 362d1b3d66794950885b3c6de5f431e0 (table test-table [id=0b88847efe604089bcec228ac10d8b73]) successfully deleted
I20260619 14:54:29.653142 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:29.616826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49636 (local address 127.0.241.1:34411)
0619 14:54:29.616985 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:29.616993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:29.634953 (+ 17960us) server_negotiation.cc:407] Connection header received
0619 14:54:29.635316 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:29.635322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:29.635516 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:29.635674 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:29.637417 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.638647 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.640547 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.641215 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.641363 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.646726 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:29.646791 (+ 65us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:29.646808 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:29.646861 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:29.650108 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:29.650987 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:29.650994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:29.650996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:29.651127 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:29.651580 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:29.651583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:29.651585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:29.651905 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:29.652134 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:29.652491 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 14:54:29.652734 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:54:29.653872 19371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:29.751696 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 19248
I20260619 14:54:29.770430 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 19382
Jun 19 14:54:29 dist-test-slave-p3ml krb5kdc[19142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:54:29.785995 19660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:29.771158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.0.241.2:37625)
0619 14:54:29.771401 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:29.771408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:29.771473 (+ 65us) server_negotiation.cc:407] Connection header received
0619 14:54:29.771702 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:29.771707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:29.771778 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:29.771901 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:29.773450 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.774248 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.775773 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.776164 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.776260 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.780589 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:29.780603 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:29.780618 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:29.780659 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:29.783399 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:29.784085 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:29.784107 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:29.784109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:29.784250 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:29.784555 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:29.784559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:29.784561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:29.784970 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:29.785200 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:29.785572 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 14:54:29.785737 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260619 14:54:29.786729 19505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:29.874845 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 19382
I20260619 14:54:29.891355 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 19515
Jun 19 14:54:29 dist-test-slave-p3ml krb5kdc[19142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:54:29.906276 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:29.892184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.0.241.3:36179)
0619 14:54:29.892361 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:29.892365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:29.892383 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:54:29.892476 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:29.892479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:29.892550 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:29.892635 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:29.893939 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.894729 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.896071 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:29.896469 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:29.896560 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:29.900937 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:29.900978 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:29.900993 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:29.901023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:29.903889 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:29.904552 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:29.904558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:29.904560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:29.904684 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:29.904993 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:29.904996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:29.904998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:29.905323 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:29.905522 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:29.905816 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 14:54:29.905972 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:54:29.906904 19638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:29.994432 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 19515
I20260619 14:54:30.011751 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 19155
I20260619 14:54:30.012907 19217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:30.121495 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 19155
2026-06-19T14:54:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4558 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T14:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:30Z Disabled control of system clock
I20260619 14:54:30.196888 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35293
--webserver_interface=127.0.241.62
--webserver_port=0
--builtin_ntp_servers=127.0.241.20:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:35293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:54:30.518553 19700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:30.518827 19700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:30.518924 19700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:30.526113 19700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:30.526213 19700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:30.526259 19700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:30.526322 19700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:30.526360 19700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:30.535897 19700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:35293
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:30.537078 19700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:30.538780 19700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:30.550671 19705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:30.550720 19706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:30.550514 19708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:30.551786 19700 server_base.cc:1034] running on GCE node
I20260619 14:54:30.552654 19700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:30.554008 19700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:30.555244 19700 hybrid_clock.cc:648] HybridClock initialized: now 1781880870555188 us; error 47 us; skew 500 ppm
I20260619 14:54:30.555639 19700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:54:30.560679 19700 webserver.cc:460] Webserver started at http://127.0.241.62:33643/ using document root <none> and password file <none>
I20260619 14:54:30.561350 19700 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:30.561486 19700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:30.561805 19700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:30.564744 19700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "37245f78e6f84892a870113b1243182e"
format_stamp: "Formatted at 2026-06-19 14:54:30 on dist-test-slave-p3ml"
server_key: "bb09cc4e91a60916de9593e1ea97e703"
server_key_iv: "4d9ed394d8cbbaf714334cf045a7d3e6"
server_key_version: "encryptionkey@0"
I20260619 14:54:30.565505 19700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "37245f78e6f84892a870113b1243182e"
format_stamp: "Formatted at 2026-06-19 14:54:30 on dist-test-slave-p3ml"
server_key: "bb09cc4e91a60916de9593e1ea97e703"
server_key_iv: "4d9ed394d8cbbaf714334cf045a7d3e6"
server_key_version: "encryptionkey@0"
I20260619 14:54:30.572573 19700 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260619 14:54:30.577385 19714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:30.579262 19700 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:54:30.579458 19700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "37245f78e6f84892a870113b1243182e"
format_stamp: "Formatted at 2026-06-19 14:54:30 on dist-test-slave-p3ml"
server_key: "bb09cc4e91a60916de9593e1ea97e703"
server_key_iv: "4d9ed394d8cbbaf714334cf045a7d3e6"
server_key_version: "encryptionkey@0"
I20260619 14:54:30.579710 19700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:30.601469 19700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:30.602197 19700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:30.602432 19700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:30.619849 19700 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:35293
I20260619 14:54:30.619884 19765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:35293 every 8 connection(s)
I20260619 14:54:30.621088 19700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:30.624625 19766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:30.626524 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 19700
I20260619 14:54:30.626716 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:30.627298 964 external_mini_cluster.cc:1428] Setting key 9123e664bb8c233cf4bfb9cbc0bdcd29
I20260619 14:54:30.636224 19766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e: Bootstrap starting.
I20260619 14:54:30.639422 19766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:30.640487 19766 log.cc:824] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:30.640529 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:30.629280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.0.241.62:35293)
0619 14:54:30.630223 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:30.630232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:30.630271 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:54:30.632031 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:30.632052 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:30.632497 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:30.632856 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:30.633435 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:30.634779 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:30.636234 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:30.636948 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:30.637148 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:30.638315 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:30.638322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:30.638334 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:30.638825 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:30.638909 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:30.638932 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:30.639053 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:30.639276 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 14:54:30.639635 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":358,"threads_started":1}
I20260619 14:54:30.643290 19766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e: No bootstrap required, opened a new log
I20260619 14:54:30.650785 19766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37245f78e6f84892a870113b1243182e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35293 } }
I20260619 14:54:30.651265 19766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:30.651396 19766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37245f78e6f84892a870113b1243182e, State: Initialized, Role: FOLLOWER
I20260619 14:54:30.652004 19766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37245f78e6f84892a870113b1243182e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35293 } }
I20260619 14:54:30.652261 19766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:30.652415 19766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:30.652581 19766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:30.654708 19766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37245f78e6f84892a870113b1243182e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35293 } }
I20260619 14:54:30.655210 19766 leader_election.cc:302] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37245f78e6f84892a870113b1243182e; no voters:
I20260619 14:54:30.655808 19766 leader_election.cc:288] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:30.655938 19771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:30.656678 19771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [term 1 LEADER]: Becoming Leader. State: Replica: 37245f78e6f84892a870113b1243182e, State: Running, Role: LEADER
I20260619 14:54:30.657137 19771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37245f78e6f84892a870113b1243182e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35293 } }
I20260619 14:54:30.657963 19766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:30.659765 19773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37245f78e6f84892a870113b1243182e. Latest consensus state: current_term: 1 leader_uuid: "37245f78e6f84892a870113b1243182e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37245f78e6f84892a870113b1243182e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35293 } } }
I20260619 14:54:30.660118 19773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:30.660477 19772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37245f78e6f84892a870113b1243182e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37245f78e6f84892a870113b1243182e" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35293 } } }
I20260619 14:54:30.660848 19772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:30.662782 19776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:30.667761 19776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:30.676362 19776 catalog_manager.cc:1347] Generated new cluster ID: dc028bebd7324cea8c9a7b5b7d7c0ebd
I20260619 14:54:30.676472 19776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:30.696535 19776 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:30.697824 19776 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:30.707768 19776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e: Generated new TSK 0
I20260619 14:54:30.708793 19776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:30.721462 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35293
--builtin_ntp_servers=127.0.241.20:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:54:31.050040 19790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:31.050369 19790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:31.050530 19790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:31.057884 19790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:31.058012 19790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:31.058179 19790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:54:31.067775 19790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:31.069090 19790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:31.070886 19790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:31.084616 19795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:31.084585 19798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:31.084694 19796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:31.085474 19790 server_base.cc:1034] running on GCE node
I20260619 14:54:31.086073 19790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:31.086993 19790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:31.088230 19790 hybrid_clock.cc:648] HybridClock initialized: now 1781880871088177 us; error 59 us; skew 500 ppm
I20260619 14:54:31.088557 19790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:54:31.090807 19790 webserver.cc:460] Webserver started at http://127.0.241.1:41375/ using document root <none> and password file <none>
I20260619 14:54:31.091578 19790 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:31.091710 19790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:31.092036 19790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:31.094856 19790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "1986339e49964aef819a8b245de610cc"
format_stamp: "Formatted at 2026-06-19 14:54:31 on dist-test-slave-p3ml"
server_key: "1979134f443a24f190209818c6c06b4e"
server_key_iv: "c0f8e0729ba90b9b414783dd25c2a12e"
server_key_version: "encryptionkey@0"
I20260619 14:54:31.095731 19790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "1986339e49964aef819a8b245de610cc"
format_stamp: "Formatted at 2026-06-19 14:54:31 on dist-test-slave-p3ml"
server_key: "1979134f443a24f190209818c6c06b4e"
server_key_iv: "c0f8e0729ba90b9b414783dd25c2a12e"
server_key_version: "encryptionkey@0"
I20260619 14:54:31.100682 19790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:54:31.104001 19804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:31.105342 19790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:54:31.105530 19790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "1986339e49964aef819a8b245de610cc"
format_stamp: "Formatted at 2026-06-19 14:54:31 on dist-test-slave-p3ml"
server_key: "1979134f443a24f190209818c6c06b4e"
server_key_iv: "c0f8e0729ba90b9b414783dd25c2a12e"
server_key_version: "encryptionkey@0"
I20260619 14:54:31.105746 19790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:31.122745 19790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:31.123608 19790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:31.123890 19790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:31.124888 19790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:31.126272 19790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:31.126405 19790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:31.126534 19790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:31.126624 19790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:31.160233 19790 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:45593
I20260619 14:54:31.160279 19916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:45593 every 8 connection(s)
I20260619 14:54:31.161705 19790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:54:31.166079 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 19790
I20260619 14:54:31.166231 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:54:31.166712 964 external_mini_cluster.cc:1428] Setting key 335339656e100edbba0ab232ecea4164
I20260619 14:54:31.169266 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35293
--builtin_ntp_servers=127.0.241.20:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:54:31.174568 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:31.164548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:60005 (local address 127.0.241.62:35293)
0619 14:54:31.164998 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:31.165004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:31.166229 (+ 1225us) server_negotiation.cc:407] Connection header received
0619 14:54:31.168009 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:31.168014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:31.168102 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:31.168216 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:31.169257 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:31.170215 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:31.171818 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:31.172241 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:31.172371 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:31.173706 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:31.173712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:31.173717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:31.173737 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:31.173789 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:31.173800 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:31.173888 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:31.174119 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 14:54:31.174214 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":219,"threads_started":1}
I20260619 14:54:31.175585 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:31.165164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35293 (local address 127.0.241.1:60005)
0619 14:54:31.166022 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:31.166060 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:31.167748 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:31.168439 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:31.168447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:31.168529 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:31.169085 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:31.169102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:31.170373 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:31.170378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:31.171676 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:31.171687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:31.172458 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:31.172465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:31.172691 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:31.173411 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:54:31.173413 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:31.173450 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:54:31.173523 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:31.173903 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:31.173916 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:31.173928 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 14:54:31.174102 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 14:54:31.174474 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":259,"threads_started":1}
I20260619 14:54:31.177037 19917 heartbeater.cc:347] Connected to a master server at 127.0.241.62:35293
I20260619 14:54:31.177326 19917 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:31.177999 19917 heartbeater.cc:511] Master 127.0.241.62:35293 requested a full tablet report, sending...
I20260619 14:54:31.179991 19731 ts_manager.cc:194] Registered new tserver with Master: 1986339e49964aef819a8b245de610cc (127.0.241.1:45593)
I20260619 14:54:31.181416 19731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.241.1:60005
W20260619 14:54:31.498561 19921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:31.498822 19921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:31.498956 19921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:31.506273 19921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:31.506372 19921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:31.506549 19921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:54:31.515849 19921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:31.517161 19921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:31.518905 19921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:31.531781 19929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:31.531849 19927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:31.531890 19926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:31.532776 19921 server_base.cc:1034] running on GCE node
I20260619 14:54:31.533329 19921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:31.534191 19921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:31.535406 19921 hybrid_clock.cc:648] HybridClock initialized: now 1781880871535367 us; error 50 us; skew 500 ppm
I20260619 14:54:31.535737 19921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:54:31.537786 19921 webserver.cc:460] Webserver started at http://127.0.241.2:44107/ using document root <none> and password file <none>
I20260619 14:54:31.538425 19921 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:31.538544 19921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:31.538843 19921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:31.541561 19921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "4097aa5fa08f4c01bd7f3d462ef30b43"
format_stamp: "Formatted at 2026-06-19 14:54:31 on dist-test-slave-p3ml"
server_key: "bd95466a47ae12ac23201df51c17b505"
server_key_iv: "be0c3361a081b0f3a69f050b8832bf95"
server_key_version: "encryptionkey@0"
I20260619 14:54:31.542300 19921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "4097aa5fa08f4c01bd7f3d462ef30b43"
format_stamp: "Formatted at 2026-06-19 14:54:31 on dist-test-slave-p3ml"
server_key: "bd95466a47ae12ac23201df51c17b505"
server_key_iv: "be0c3361a081b0f3a69f050b8832bf95"
server_key_version: "encryptionkey@0"
I20260619 14:54:31.546942 19921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 14:54:31.550114 19935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:31.551381 19921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:31.551558 19921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "4097aa5fa08f4c01bd7f3d462ef30b43"
format_stamp: "Formatted at 2026-06-19 14:54:31 on dist-test-slave-p3ml"
server_key: "bd95466a47ae12ac23201df51c17b505"
server_key_iv: "be0c3361a081b0f3a69f050b8832bf95"
server_key_version: "encryptionkey@0"
I20260619 14:54:31.551764 19921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:31.565310 19921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:31.566023 19921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:31.566284 19921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:31.567502 19921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:31.570457 19921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:31.570566 19921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:31.570674 19921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:31.570755 19921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:31.603986 19921 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:37759
I20260619 14:54:31.604043 20047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:37759 every 8 connection(s)
I20260619 14:54:31.605304 19921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:54:31.613641 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 19921
I20260619 14:54:31.613811 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:54:31.614282 964 external_mini_cluster.cc:1428] Setting key 97bf6c406d843886090a37df363d9f2f
I20260619 14:54:31.616871 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35293
--builtin_ntp_servers=127.0.241.20:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:54:31.617589 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:31.607824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:38265 (local address 127.0.241.62:35293)
0619 14:54:31.608027 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:31.608034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:31.609473 (+ 1439us) server_negotiation.cc:407] Connection header received
0619 14:54:31.611261 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:31.611265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:31.611330 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:31.611410 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:31.612408 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:31.613095 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:31.614741 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:31.615292 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:31.615424 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:31.616808 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:31.616814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:31.616819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:31.616839 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:31.616875 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:31.616887 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:31.616986 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:31.617181 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 14:54:31.617273 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:54:31.618536 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:31.608364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35293 (local address 127.0.241.2:38265)
0619 14:54:31.609328 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:31.609368 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:31.611010 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:31.611642 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:31.611650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:31.611729 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:31.612273 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:31.612288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:31.613230 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:31.613234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:31.614594 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:31.614606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:31.615542 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:31.615551 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:31.615781 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:31.616522 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:54:31.616525 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:31.616560 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:54:31.616629 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:31.616976 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:31.616986 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:31.616995 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:54:31.617171 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 14:54:31.617536 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":349,"threads_started":1}
I20260619 14:54:31.619907 20048 heartbeater.cc:347] Connected to a master server at 127.0.241.62:35293
I20260619 14:54:31.620200 20048 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:31.620882 20048 heartbeater.cc:511] Master 127.0.241.62:35293 requested a full tablet report, sending...
I20260619 14:54:31.622200 19731 ts_manager.cc:194] Registered new tserver with Master: 4097aa5fa08f4c01bd7f3d462ef30b43 (127.0.241.2:37759)
I20260619 14:54:31.623009 19731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.241.2:38265
W20260619 14:54:31.937443 20051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:31.937716 20051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:31.937803 20051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:31.944923 20051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:31.945022 20051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:31.945153 20051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:54:31.954041 20051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:31.955366 20051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:31.957149 20051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:31.969727 20056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:31.969821 20057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:31.969857 20059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:31.970660 20051 server_base.cc:1034] running on GCE node
I20260619 14:54:31.971293 20051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:31.972206 20051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:31.973433 20051 hybrid_clock.cc:648] HybridClock initialized: now 1781880871973393 us; error 36 us; skew 500 ppm
I20260619 14:54:31.973765 20051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:54:31.976089 20051 webserver.cc:460] Webserver started at http://127.0.241.3:42885/ using document root <none> and password file <none>
I20260619 14:54:31.976763 20051 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:31.976871 20051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:31.977200 20051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:31.980530 20051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "6e3f756f17424d23a15a84bcee5f736e"
format_stamp: "Formatted at 2026-06-19 14:54:31 on dist-test-slave-p3ml"
server_key: "729fbb1c7b2498ffcd825647c597d082"
server_key_iv: "7323862b34c6184ad8554d9694fa3a15"
server_key_version: "encryptionkey@0"
I20260619 14:54:31.981292 20051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "6e3f756f17424d23a15a84bcee5f736e"
format_stamp: "Formatted at 2026-06-19 14:54:31 on dist-test-slave-p3ml"
server_key: "729fbb1c7b2498ffcd825647c597d082"
server_key_iv: "7323862b34c6184ad8554d9694fa3a15"
server_key_version: "encryptionkey@0"
I20260619 14:54:31.986191 20051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:54:31.989378 20065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:31.990573 20051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:31.990744 20051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "6e3f756f17424d23a15a84bcee5f736e"
format_stamp: "Formatted at 2026-06-19 14:54:31 on dist-test-slave-p3ml"
server_key: "729fbb1c7b2498ffcd825647c597d082"
server_key_iv: "7323862b34c6184ad8554d9694fa3a15"
server_key_version: "encryptionkey@0"
I20260619 14:54:31.990950 20051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:32.009367 20051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:32.010176 20051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:32.010442 20051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:32.011364 20051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:32.012735 20051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:32.012858 20051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:32.012987 20051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:32.013077 20051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:32.044862 20051 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:35207
I20260619 14:54:32.044919 20177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:35207 every 8 connection(s)
I20260619 14:54:32.046183 20051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:54:32.053043 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 20051
I20260619 14:54:32.053224 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:54:32.053730 964 external_mini_cluster.cc:1428] Setting key 58b59136510eb2d5e7a87c6defbdfaa8
I20260619 14:54:32.059018 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.048648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:44015 (local address 127.0.241.62:35293)
0619 14:54:32.048858 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:32.048865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:32.050243 (+ 1378us) server_negotiation.cc:407] Connection header received
0619 14:54:32.052060 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:32.052065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:32.052133 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:32.052232 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:32.053380 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.054339 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.056420 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.056909 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.057004 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.058231 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:32.058236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:32.058239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:32.058256 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:32.058285 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:32.058296 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:32.058396 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:32.058611 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 14:54:32.058706 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:54:32.060003 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.049151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35293 (local address 127.0.241.3:44015)
0619 14:54:32.050096 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:32.050131 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:32.051808 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:32.052436 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:32.052444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:32.052534 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:32.053157 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:32.053184 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.054516 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.054521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:32.056289 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:32.056304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.057087 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.057092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:32.057288 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.057939 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:54:32.057941 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:32.057979 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:54:32.058059 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:32.058400 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:32.058412 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:32.058422 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:54:32.058599 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 14:54:32.058979 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":318,"threads_started":1}
I20260619 14:54:32.061196 20178 heartbeater.cc:347] Connected to a master server at 127.0.241.62:35293
I20260619 14:54:32.061491 20178 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:32.062229 20178 heartbeater.cc:511] Master 127.0.241.62:35293 requested a full tablet report, sending...
I20260619 14:54:32.063622 19731 ts_manager.cc:194] Registered new tserver with Master: 6e3f756f17424d23a15a84bcee5f736e (127.0.241.3:35207)
I20260619 14:54:32.064420 19731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.241.3:44015
I20260619 14:54:32.068451 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:54:32.078084 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.072326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35806 (local address 127.0.241.62:35293)
0619 14:54:32.072502 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:32.072506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:32.072937 (+ 431us) server_negotiation.cc:407] Connection header received
0619 14:54:32.073061 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:32.073064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:32.073129 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:32.073217 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:32.073622 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.074283 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.075653 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.076025 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.076159 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.077376 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:32.077381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:32.077385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:32.077401 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:32.077431 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:32.077443 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:32.077582 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:32.077768 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 14:54:32.077825 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:54:32.081635 19731 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:54:32.088470 19731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:54:32.123627 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.099501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.0.241.2:37759)
0619 14:54:32.115549 (+ 16048us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:32.115559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:32.115588 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:54:32.115665 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:32.115671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:32.115901 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:32.116082 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:32.117145 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.118187 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.120983 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.122286 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.122473 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.122733 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:32.123061 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 14:54:32.123232 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15856,"thread_start_us":15723,"threads_started":1}
I20260619 14:54:32.123694 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.099357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:37759 (local address 127.0.0.1:50710)
0619 14:54:32.112130 (+ 12773us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:32.112183 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:32.112384 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:32.116346 (+ 3962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:32.116350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:32.116375 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:32.116941 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:32.116954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.118321 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.118328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:32.120775 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:32.120791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.122519 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.122545 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:32.122780 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.122834 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 14:54:32.122984 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 14:54:32.123172 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12469,"mutex_wait_us":53,"thread_start_us":16279,"threads_started":1}
I20260619 14:54:32.123733 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.099742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45593 (local address 127.0.0.1:35798)
0619 14:54:32.115177 (+ 15435us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:32.115198 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:32.115411 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:32.116228 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:32.116234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:32.116262 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:32.116791 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:32.116810 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.118737 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.118743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:32.120979 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:32.120995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.122822 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.122828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:32.122998 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.123030 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:54:32.123123 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:54:32.123272 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15301,"mutex_wait_us":54,"thread_start_us":12834,"threads_started":1}
I20260619 14:54:32.124459 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.100178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:35207 (local address 127.0.0.1:35746)
0619 14:54:32.115883 (+ 15705us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:32.115903 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:32.116037 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:32.117396 (+ 1359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:32.117403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:32.117428 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:32.117745 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:32.117769 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.119416 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.119422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:32.121595 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:32.121610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.123681 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.123689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:32.123897 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.123927 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:54:32.124010 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 14:54:32.124126 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15594,"mutex_wait_us":1,"thread_start_us":14715,"threads_started":1}
I20260619 14:54:32.124683 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.100030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.0.241.1:45593)
0619 14:54:32.109029 (+ 8999us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:32.109042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:32.115303 (+ 6261us) server_negotiation.cc:407] Connection header received
0619 14:54:32.115640 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:32.115648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:32.115904 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:32.116070 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:32.117024 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.118602 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.121148 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.122305 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.122432 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.122686 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:32.124127 (+ 1441us) server_negotiation.cc:299] Negotiation successful
0619 14:54:32.124304 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8794,"thread_start_us":8632,"threads_started":1}
I20260619 14:54:32.125263 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.100802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35746 (local address 127.0.241.3:35207)
0619 14:54:32.114684 (+ 13882us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:32.114706 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:32.116003 (+ 1297us) server_negotiation.cc:407] Connection header received
0619 14:54:32.116781 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:32.116789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:32.117021 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:32.117213 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:32.118093 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.119229 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.121790 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.123138 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.123252 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.123518 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:32.124734 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0619 14:54:32.124903 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13682,"thread_start_us":13529,"threads_started":1}
I20260619 14:54:32.128343 20113 tablet_service.cc:1467] Processing CreateTablet for tablet ce51596d18fb4f5184c652fd1ff97fea (DEFAULT_TABLE table=test-table [id=ffa3a13e5a144d4e80d3e69b6eb75d29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:32.128386 19852 tablet_service.cc:1467] Processing CreateTablet for tablet ce51596d18fb4f5184c652fd1ff97fea (DEFAULT_TABLE table=test-table [id=ffa3a13e5a144d4e80d3e69b6eb75d29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:32.128386 19983 tablet_service.cc:1467] Processing CreateTablet for tablet ce51596d18fb4f5184c652fd1ff97fea (DEFAULT_TABLE table=test-table [id=ffa3a13e5a144d4e80d3e69b6eb75d29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:32.129738 19852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce51596d18fb4f5184c652fd1ff97fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:32.129752 19983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce51596d18fb4f5184c652fd1ff97fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:32.129805 20113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce51596d18fb4f5184c652fd1ff97fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:32.141122 20196 tablet_bootstrap.cc:492] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e: Bootstrap starting.
I20260619 14:54:32.145192 20196 tablet_bootstrap.cc:654] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:32.146095 20198 tablet_bootstrap.cc:492] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc: Bootstrap starting.
I20260619 14:54:32.146596 20196 log.cc:824] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:32.147045 20197 tablet_bootstrap.cc:492] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43: Bootstrap starting.
I20260619 14:54:32.150194 20198 tablet_bootstrap.cc:654] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:32.150231 20196 tablet_bootstrap.cc:492] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e: No bootstrap required, opened a new log
I20260619 14:54:32.150576 20196 ts_tablet_manager.cc:1397] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 14:54:32.150959 20197 tablet_bootstrap.cc:654] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:32.151432 20198 log.cc:824] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:32.152267 20197 log.cc:824] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:32.155361 20197 tablet_bootstrap.cc:492] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43: No bootstrap required, opened a new log
I20260619 14:54:32.155381 20198 tablet_bootstrap.cc:492] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc: No bootstrap required, opened a new log
I20260619 14:54:32.155663 20198 ts_tablet_manager.cc:1397] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 14:54:32.155668 20197 ts_tablet_manager.cc:1397] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 14:54:32.161710 20196 raft_consensus.cc:348] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37759 } } peers { permanent_uuid: "1986339e49964aef819a8b245de610cc" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45593 } } peers { permanent_uuid: "6e3f756f17424d23a15a84bcee5f736e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 35207 } }
I20260619 14:54:32.162235 20196 raft_consensus.cc:374] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:32.162326 20196 raft_consensus.cc:729] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3f756f17424d23a15a84bcee5f736e, State: Initialized, Role: FOLLOWER
I20260619 14:54:32.163033 20196 consensus_queue.cc:260] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37759 } } peers { permanent_uuid: "1986339e49964aef819a8b245de610cc" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45593 } } peers { permanent_uuid: "6e3f756f17424d23a15a84bcee5f736e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 35207 } }
I20260619 14:54:32.164891 20197 raft_consensus.cc:348] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37759 } } peers { permanent_uuid: "1986339e49964aef819a8b245de610cc" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45593 } } peers { permanent_uuid: "6e3f756f17424d23a15a84bcee5f736e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 35207 } }
I20260619 14:54:32.165282 20197 raft_consensus.cc:374] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:32.165396 20197 raft_consensus.cc:729] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4097aa5fa08f4c01bd7f3d462ef30b43, State: Initialized, Role: FOLLOWER
I20260619 14:54:32.165943 20197 consensus_queue.cc:260] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37759 } } peers { permanent_uuid: "1986339e49964aef819a8b245de610cc" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45593 } } peers { permanent_uuid: "6e3f756f17424d23a15a84bcee5f736e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 35207 } }
I20260619 14:54:32.171036 20196 ts_tablet_manager.cc:1428] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e: Time spent starting tablet: real 0.020s user 0.021s sys 0.000s
I20260619 14:54:32.171358 20178 heartbeater.cc:503] Master 127.0.241.62:35293 was elected leader, sending a full tablet report...
I20260619 14:54:32.173991 20198 raft_consensus.cc:348] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37759 } } peers { permanent_uuid: "1986339e49964aef819a8b245de610cc" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45593 } } peers { permanent_uuid: "6e3f756f17424d23a15a84bcee5f736e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 35207 } }
I20260619 14:54:32.174620 20198 raft_consensus.cc:374] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:32.174806 20198 raft_consensus.cc:729] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1986339e49964aef819a8b245de610cc, State: Initialized, Role: FOLLOWER
I20260619 14:54:32.176692 20198 consensus_queue.cc:260] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37759 } } peers { permanent_uuid: "1986339e49964aef819a8b245de610cc" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45593 } } peers { permanent_uuid: "6e3f756f17424d23a15a84bcee5f736e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 35207 } }
I20260619 14:54:32.178203 20048 heartbeater.cc:503] Master 127.0.241.62:35293 was elected leader, sending a full tablet report...
I20260619 14:54:32.178936 20197 ts_tablet_manager.cc:1428] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43: Time spent starting tablet: real 0.023s user 0.014s sys 0.007s
I20260619 14:54:32.178956 19917 heartbeater.cc:503] Master 127.0.241.62:35293 was elected leader, sending a full tablet report...
I20260619 14:54:32.179528 20198 ts_tablet_manager.cc:1428] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc: Time spent starting tablet: real 0.024s user 0.016s sys 0.000s
I20260619 14:54:32.226552 20202 raft_consensus.cc:482] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:32.226887 20202 raft_consensus.cc:504] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37759 } } peers { permanent_uuid: "1986339e49964aef819a8b245de610cc" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45593 } } peers { permanent_uuid: "6e3f756f17424d23a15a84bcee5f736e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 35207 } }
I20260619 14:54:32.228610 20202 leader_election.cc:288] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4097aa5fa08f4c01bd7f3d462ef30b43 (127.0.241.2:37759), 1986339e49964aef819a8b245de610cc (127.0.241.1:45593)
I20260619 14:54:32.234864 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.228959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45593 (local address 127.0.241.3:50377)
0619 14:54:32.229311 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:32.229336 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:32.229504 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:32.229954 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:32.229963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:32.229985 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:32.230191 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:32.230205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.231378 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.231383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:32.232897 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:32.232909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.234133 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.234139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:32.234323 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.234361 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:54:32.234459 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:54:32.234599 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260619 14:54:32.234885 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.229077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:50377 (local address 127.0.241.1:45593)
0619 14:54:32.229270 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:32.229278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:32.229475 (+ 197us) server_negotiation.cc:407] Connection header received
0619 14:54:32.229669 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:32.229674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:32.229770 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:32.229890 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:32.230341 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.231240 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.233052 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.233985 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.234081 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.234144 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:32.234479 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 14:54:32.234618 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:54:32.235342 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.229201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:51385 (local address 127.0.241.2:37759)
0619 14:54:32.229415 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:32.229422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:32.229914 (+ 492us) server_negotiation.cc:407] Connection header received
0619 14:54:32.230076 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:32.230081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:32.230168 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:32.230269 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:32.230866 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.231692 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.233535 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.234438 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.234548 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.234626 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:32.234982 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 14:54:32.235115 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:54:32.235606 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.228994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:37759 (local address 127.0.241.3:51385)
0619 14:54:32.229797 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:32.229816 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:32.229942 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:32.230299 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:32.230303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:32.230329 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:32.230684 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:32.230697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.231833 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.231837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:32.233352 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:32.233367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.234594 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.234600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:32.234803 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.234835 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:54:32.235076 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 14:54:32.235210 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":444,"threads_started":1}
I20260619 14:54:32.235924 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce51596d18fb4f5184c652fd1ff97fea" candidate_uuid: "6e3f756f17424d23a15a84bcee5f736e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1986339e49964aef819a8b245de610cc" is_pre_election: true
I20260619 14:54:32.236526 19872 raft_consensus.cc:2454] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3f756f17424d23a15a84bcee5f736e in term 0.
I20260619 14:54:32.236606 20003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce51596d18fb4f5184c652fd1ff97fea" candidate_uuid: "6e3f756f17424d23a15a84bcee5f736e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43" is_pre_election: true
I20260619 14:54:32.237116 20003 raft_consensus.cc:2454] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e3f756f17424d23a15a84bcee5f736e in term 0.
I20260619 14:54:32.237210 20067 leader_election.cc:302] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1986339e49964aef819a8b245de610cc, 6e3f756f17424d23a15a84bcee5f736e; no voters:
I20260619 14:54:32.237627 20202 raft_consensus.cc:2781] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:54:32.237802 20202 raft_consensus.cc:482] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:54:32.237906 20202 raft_consensus.cc:3037] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:32.240031 20202 raft_consensus.cc:504] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37759 } } peers { permanent_uuid: "1986339e49964aef819a8b245de610cc" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45593 } } peers { permanent_uuid: "6e3f756f17424d23a15a84bcee5f736e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 35207 } }
I20260619 14:54:32.240578 20202 leader_election.cc:288] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [CANDIDATE]: Term 1 election: Requested vote from peers 4097aa5fa08f4c01bd7f3d462ef30b43 (127.0.241.2:37759), 1986339e49964aef819a8b245de610cc (127.0.241.1:45593)
I20260619 14:54:32.240937 20003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce51596d18fb4f5184c652fd1ff97fea" candidate_uuid: "6e3f756f17424d23a15a84bcee5f736e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43"
I20260619 14:54:32.241142 20003 raft_consensus.cc:3037] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:32.241190 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce51596d18fb4f5184c652fd1ff97fea" candidate_uuid: "6e3f756f17424d23a15a84bcee5f736e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1986339e49964aef819a8b245de610cc"
I20260619 14:54:32.241407 19872 raft_consensus.cc:3037] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:32.243368 20003 raft_consensus.cc:2454] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3f756f17424d23a15a84bcee5f736e in term 1.
I20260619 14:54:32.243719 19872 raft_consensus.cc:2454] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e3f756f17424d23a15a84bcee5f736e in term 1.
I20260619 14:54:32.243770 20066 leader_election.cc:302] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4097aa5fa08f4c01bd7f3d462ef30b43, 6e3f756f17424d23a15a84bcee5f736e; no voters:
I20260619 14:54:32.244138 20202 raft_consensus.cc:2781] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:32.244540 20202 raft_consensus.cc:686] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 1 LEADER]: Becoming Leader. State: Replica: 6e3f756f17424d23a15a84bcee5f736e, State: Running, Role: LEADER
I20260619 14:54:32.245043 20202 consensus_queue.cc:237] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37759 } } peers { permanent_uuid: "1986339e49964aef819a8b245de610cc" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45593 } } peers { permanent_uuid: "6e3f756f17424d23a15a84bcee5f736e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 35207 } }
I20260619 14:54:32.249630 19731 catalog_manager.cc:5495] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e3f756f17424d23a15a84bcee5f736e (127.0.241.3). New cstate: current_term: 1 leader_uuid: "6e3f756f17424d23a15a84bcee5f736e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1986339e49964aef819a8b245de610cc" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e3f756f17424d23a15a84bcee5f736e" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 35207 } health_report { overall_health: HEALTHY } } }
I20260619 14:54:32.279083 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.273422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.0.241.3:35207)
0619 14:54:32.273621 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:32.273627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:32.273647 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:54:32.273834 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:32.273839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:32.273914 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:32.274063 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:32.274447 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.275195 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.276819 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.277232 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.277369 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.278011 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:32.278491 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:54:32.278674 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:32.278806 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 14:54:32.278850 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:54:32.287070 19872 raft_consensus.cc:1261] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc [term 1 FOLLOWER]: Refusing update from remote peer 6e3f756f17424d23a15a84bcee5f736e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:32.287127 20003 raft_consensus.cc:1261] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43 [term 1 FOLLOWER]: Refusing update from remote peer 6e3f756f17424d23a15a84bcee5f736e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:32.288028 20206 consensus_queue.cc:1035] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4097aa5fa08f4c01bd7f3d462ef30b43" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:32.288388 20202 consensus_queue.cc:1035] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1986339e49964aef819a8b245de610cc" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:54:32.297912 20179 tablet.cc:2369] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:32.301244 20209 mvcc.cc:205] Tried to move back new op lower bound from 7298584052878512128 to 7298584052719738880. Current Snapshot: MvccSnapshot[applied={T|T < 7298584052878512128 or (T in {7298584052878512128})}]
I20260619 14:54:32.322184 19730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:32.322497 19730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:32.326221 19730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37245f78e6f84892a870113b1243182e: Sending DeleteTablet for 3 replicas of tablet ce51596d18fb4f5184c652fd1ff97fea
I20260619 14:54:32.327275 19983 tablet_service.cc:1514] Processing DeleteTablet for tablet ce51596d18fb4f5184c652fd1ff97fea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:32 UTC) from {username='slave'} at 127.0.0.1:50710
I20260619 14:54:32.327255 19852 tablet_service.cc:1514] Processing DeleteTablet for tablet ce51596d18fb4f5184c652fd1ff97fea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:32 UTC) from {username='slave'} at 127.0.0.1:35798
I20260619 14:54:32.328042 20113 tablet_service.cc:1514] Processing DeleteTablet for tablet ce51596d18fb4f5184c652fd1ff97fea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:32 UTC) from {username='slave'} at 127.0.0.1:35746
I20260619 14:54:32.329038 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 19790
I20260619 14:54:32.342505 20219 tablet_replica.cc:330] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc: stopping tablet replica
I20260619 14:54:32.343024 20219 raft_consensus.cc:2229] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:32.343595 20219 raft_consensus.cc:2258] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:32.346122 20219 ts_tablet_manager.cc:1905] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:32.355609 20219 ts_tablet_manager.cc:1918] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:32.355852 20219 log.cc:1196] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/ce51596d18fb4f5184c652fd1ff97fea
I20260619 14:54:32.356508 20219 ts_tablet_manager.cc:1939] T ce51596d18fb4f5184c652fd1ff97fea P 1986339e49964aef819a8b245de610cc: Deleting consensus metadata
I20260619 14:54:32.358403 19716 catalog_manager.cc:4841] TS 1986339e49964aef819a8b245de610cc (127.0.241.1:45593): tablet ce51596d18fb4f5184c652fd1ff97fea (table test-table [id=ffa3a13e5a144d4e80d3e69b6eb75d29]) successfully deleted
I20260619 14:54:32.360683 20220 tablet_replica.cc:330] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43: stopping tablet replica
I20260619 14:54:32.361135 20220 raft_consensus.cc:2229] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:32.361688 20220 raft_consensus.cc:2258] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:32.362464 20221 tablet_replica.cc:330] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e: stopping tablet replica
I20260619 14:54:32.362912 20221 raft_consensus.cc:2229] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:54:32.363979 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.330686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.0.241.1:45593)
0619 14:54:32.330923 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:32.330933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:32.354756 (+ 23823us) server_negotiation.cc:407] Connection header received
0619 14:54:32.355267 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:32.355275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:32.355389 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:32.355496 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:32.356214 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.357354 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.359507 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.360363 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.360492 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.362218 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:32.362228 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:32.362248 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:32.362986 (+ 738us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:32.363045 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:32.363072 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:32.363153 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:32.363502 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 14:54:32.363612 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:54:32.364302 20221 raft_consensus.cc:2258] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:32.365319 19912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:32.365430 20220 ts_tablet_manager.cc:1905] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:32.367200 20221 ts_tablet_manager.cc:1905] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:32.372534 20220 ts_tablet_manager.cc:1918] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:32.372756 20220 log.cc:1196] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/ce51596d18fb4f5184c652fd1ff97fea
I20260619 14:54:32.373332 20220 ts_tablet_manager.cc:1939] T ce51596d18fb4f5184c652fd1ff97fea P 4097aa5fa08f4c01bd7f3d462ef30b43: Deleting consensus metadata
I20260619 14:54:32.375162 19715 catalog_manager.cc:4841] TS 4097aa5fa08f4c01bd7f3d462ef30b43 (127.0.241.2:37759): tablet ce51596d18fb4f5184c652fd1ff97fea (table test-table [id=ffa3a13e5a144d4e80d3e69b6eb75d29]) successfully deleted
I20260619 14:54:32.376899 20221 ts_tablet_manager.cc:1918] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:32.377108 20221 log.cc:1196] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/ce51596d18fb4f5184c652fd1ff97fea
I20260619 14:54:32.377702 20221 ts_tablet_manager.cc:1939] T ce51596d18fb4f5184c652fd1ff97fea P 6e3f756f17424d23a15a84bcee5f736e: Deleting consensus metadata
I20260619 14:54:32.379415 19717 catalog_manager.cc:4841] TS 6e3f756f17424d23a15a84bcee5f736e (127.0.241.3:35207): tablet ce51596d18fb4f5184c652fd1ff97fea (table test-table [id=ffa3a13e5a144d4e80d3e69b6eb75d29]) successfully deleted
I20260619 14:54:32.462582 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 19790
I20260619 14:54:32.480646 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 19921
I20260619 14:54:32.488389 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.481481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.0.241.2:37759)
0619 14:54:32.481690 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:32.481699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:32.481720 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:54:32.481789 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:32.481794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:32.481912 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:32.482023 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:32.482373 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.483514 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.484944 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.485616 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.485757 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.486803 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:32.486813 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:32.486832 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:32.487630 (+ 798us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:32.487700 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:32.487726 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:32.487838 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:32.487918 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 14:54:32.488026 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:54:32.489123 20043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:32.576375 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 19921
I20260619 14:54:32.593837 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 20051
I20260619 14:54:32.601082 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:32.594644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35764 (local address 127.0.241.3:35207)
0619 14:54:32.594862 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:32.594869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:32.595049 (+ 180us) server_negotiation.cc:407] Connection header received
0619 14:54:32.595229 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:32.595234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:32.595314 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:32.595427 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:32.595823 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.596654 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.597944 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:32.598377 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:32.598508 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:32.599654 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:32.599662 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:32.599680 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:32.600287 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:32.600341 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:32.600360 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:32.600505 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:32.600655 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 14:54:32.600770 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:54:32.601820 20173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:32.720330 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 20051
I20260619 14:54:32.739717 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 19700
I20260619 14:54:32.740780 19761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:32.846868 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 19700
2026-06-19T14:54:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2723 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:32 dist-test-slave-p3ml krb5kdc[20229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:32 dist-test-slave-p3ml krb5kdc[20229](info): set up 2 sockets
Jun 19 14:54:32 dist-test-slave-p3ml krb5kdc[20229](info): commencing operation
krb5kdc: starting...
W20260619 14:54:33.955685 964 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:34 dist-test-slave-p3ml krb5kdc[20229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:54:34.188664 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35133
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:43891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:35133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:34.533258 20242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:34.533556 20242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:34.533702 20242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:34.540704 20242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:34.540828 20242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:34.540915 20242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:34.541010 20242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:34.541096 20242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:34.550643 20242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:43891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:35133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:34.551926 20242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:34.553221 20242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:34.564949 20247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:34.565066 20248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:34.565143 20250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:34.565992 20242 server_base.cc:1034] running on GCE node
I20260619 14:54:34.566840 20242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:34.568285 20242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:34.569504 20242 hybrid_clock.cc:648] HybridClock initialized: now 1781880874569469 us; error 44 us; skew 500 ppm
I20260619 14:54:34.569841 20242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:34 dist-test-slave-p3ml krb5kdc[20229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:34.572851 20242 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:54:34.574285 20242 webserver.cc:460] Webserver started at http://127.0.241.62:37337/ using document root <none> and password file <none>
I20260619 14:54:34.574958 20242 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:34.575100 20242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:34.575412 20242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:34.578318 20242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "34a1b2217f864c7b9b459fe517d8f08a"
format_stamp: "Formatted at 2026-06-19 14:54:34 on dist-test-slave-p3ml"
server_key: "7190fd92770f7ed771ee711fdc97b8e1"
server_key_iv: "e49a2dcff1ebc80751c966e1e3223056"
server_key_version: "encryptionkey@0"
I20260619 14:54:34.579082 20242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "34a1b2217f864c7b9b459fe517d8f08a"
format_stamp: "Formatted at 2026-06-19 14:54:34 on dist-test-slave-p3ml"
server_key: "7190fd92770f7ed771ee711fdc97b8e1"
server_key_iv: "e49a2dcff1ebc80751c966e1e3223056"
server_key_version: "encryptionkey@0"
I20260619 14:54:34.584079 20242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 14:54:34.587286 20257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:34.588537 20242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 14:54:34.588711 20242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "34a1b2217f864c7b9b459fe517d8f08a"
format_stamp: "Formatted at 2026-06-19 14:54:34 on dist-test-slave-p3ml"
server_key: "7190fd92770f7ed771ee711fdc97b8e1"
server_key_iv: "e49a2dcff1ebc80751c966e1e3223056"
server_key_version: "encryptionkey@0"
I20260619 14:54:34.588901 20242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:34.601792 20242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:34.605504 20242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:34.605782 20242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:34.623689 20242 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:35133
I20260619 14:54:34.623718 20308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:35133 every 8 connection(s)
I20260619 14:54:34.625026 20242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:34.628469 20309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:34.630558 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 20242
I20260619 14:54:34.630729 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:34.631279 964 external_mini_cluster.cc:1428] Setting key 5bbad7b85d2554fd5bc45b35f6bd92cb
I20260619 14:54:34.640663 20309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a: Bootstrap starting.
I20260619 14:54:34.644436 20309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:34.645676 20309 log.cc:824] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:54:34 dist-test-slave-p3ml krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:54:34.648547 20309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a: No bootstrap required, opened a new log
I20260619 14:54:34.655401 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:34.633169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.0.241.62:35133)
0619 14:54:34.633960 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:34.633971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:34.634011 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:54:34.635725 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:34.635746 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:34.636094 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:34.636395 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:34.638151 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:34.639498 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:34.641337 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:34.641988 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:34.642130 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:34.647503 (+ 5373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:34.647512 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:34.647527 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:34.647570 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:34.650932 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:34.651767 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:34.651774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:34.651791 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:34.652039 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:34.652454 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:34.652460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:34.652464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:34.653037 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:34.653404 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:34.653843 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 14:54:34.654282 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":367,"threads_started":1}
I20260619 14:54:34.656454 20309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a1b2217f864c7b9b459fe517d8f08a" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35133 } }
I20260619 14:54:34.656841 20309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:34.656968 20309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a1b2217f864c7b9b459fe517d8f08a, State: Initialized, Role: FOLLOWER
I20260619 14:54:34.657497 20309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a1b2217f864c7b9b459fe517d8f08a" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35133 } }
I20260619 14:54:34.657747 20309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:34.657889 20309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:34.658061 20309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:34.660632 20309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a1b2217f864c7b9b459fe517d8f08a" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35133 } }
I20260619 14:54:34.661118 20309 leader_election.cc:302] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34a1b2217f864c7b9b459fe517d8f08a; no voters:
I20260619 14:54:34.661659 20309 leader_election.cc:288] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:34.661831 20314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:34.662636 20314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [term 1 LEADER]: Becoming Leader. State: Replica: 34a1b2217f864c7b9b459fe517d8f08a, State: Running, Role: LEADER
I20260619 14:54:34.663257 20314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a1b2217f864c7b9b459fe517d8f08a" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35133 } }
I20260619 14:54:34.663836 20309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:34.665913 20316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34a1b2217f864c7b9b459fe517d8f08a. Latest consensus state: current_term: 1 leader_uuid: "34a1b2217f864c7b9b459fe517d8f08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a1b2217f864c7b9b459fe517d8f08a" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35133 } } }
I20260619 14:54:34.666334 20316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:34.666539 20315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34a1b2217f864c7b9b459fe517d8f08a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34a1b2217f864c7b9b459fe517d8f08a" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35133 } } }
I20260619 14:54:34.666800 20315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:34.679275 20323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:34.684705 20323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:34.693730 20323 catalog_manager.cc:1347] Generated new cluster ID: 1df8c057c98c4dac9b904a6f1851bad9
I20260619 14:54:34.693847 20323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:34.711242 20323 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:34.712556 20323 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:34.725342 20323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a: Generated new TSK 0
I20260619 14:54:34.726145 20323 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.241.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
Entry for principal kudu/127.0.241.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab.
I20260619 14:54:34.796855 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35133
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:43891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:35.134080 20336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:35.134358 20336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:35.134480 20336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:35.141527 20336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:35.141633 20336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:35.141796 20336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:54:35.151474 20336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:43891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.1.keytab
--principal=kudu/127.0.241.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:35.152806 20336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:35.154109 20336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:35.168524 20342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:35.168567 20344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:35.168609 20341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:35.169705 20336 server_base.cc:1034] running on GCE node
I20260619 14:54:35.170315 20336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:35.171288 20336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:35.172453 20336 hybrid_clock.cc:648] HybridClock initialized: now 1781880875172421 us; error 52 us; skew 500 ppm
I20260619 14:54:35.172859 20336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:35 dist-test-slave-p3ml krb5kdc[20229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:35.175928 20336 init.cc:377] Logged in from keytab as kudu/127.0.241.1@KRBTEST.COM (short username kudu)
I20260619 14:54:35.177537 20336 webserver.cc:460] Webserver started at http://127.0.241.1:39797/ using document root <none> and password file <none>
I20260619 14:54:35.178262 20336 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:35.178406 20336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:35.178726 20336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:35.182156 20336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff"
format_stamp: "Formatted at 2026-06-19 14:54:35 on dist-test-slave-p3ml"
server_key: "c574932f44c499aa1faba9634cf7bc3f"
server_key_iv: "c4a902319091af42a3664ca72428d2f1"
server_key_version: "encryptionkey@0"
I20260619 14:54:35.183135 20336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff"
format_stamp: "Formatted at 2026-06-19 14:54:35 on dist-test-slave-p3ml"
server_key: "c574932f44c499aa1faba9634cf7bc3f"
server_key_iv: "c4a902319091af42a3664ca72428d2f1"
server_key_version: "encryptionkey@0"
I20260619 14:54:35.188436 20336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 14:54:35.191869 20351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:35.193495 20336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:54:35.193676 20336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff"
format_stamp: "Formatted at 2026-06-19 14:54:35 on dist-test-slave-p3ml"
server_key: "c574932f44c499aa1faba9634cf7bc3f"
server_key_iv: "c4a902319091af42a3664ca72428d2f1"
server_key_version: "encryptionkey@0"
I20260619 14:54:35.193861 20336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:35.208379 20336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:35.212821 20336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:35.213112 20336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:35.214059 20336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:35.215255 20336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:35.215338 20336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:35.215421 20336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:35.215528 20336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 14:54:35.245898 20336 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:45377
I20260619 14:54:35.245931 20463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:45377 every 8 connection(s)
I20260619 14:54:35.247174 20336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:54:35.253880 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 20336
I20260619 14:54:35.254056 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:54:35.254590 964 external_mini_cluster.cc:1428] Setting key ef5eb9056eeeb3803581834966dd9615
Jun 19 14:54:35 dist-test-slave-p3ml krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.1@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:54:35.269428 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:35.249891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:47317 (local address 127.0.241.62:35133)
0619 14:54:35.250364 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:35.250370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:35.251222 (+ 852us) server_negotiation.cc:407] Connection header received
0619 14:54:35.253013 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:35.253017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:35.253079 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:35.253171 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:35.255326 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:35.256398 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:35.258834 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:35.259218 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:35.259325 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:35.263303 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:35.263308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:35.263311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:35.263341 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:35.266570 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:35.267209 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:35.267216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:35.267220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:35.267336 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:35.267783 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:35.267788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:35.267792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:35.268039 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:35.268185 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:35.268865 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 14:54:35.269086 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":217,"threads_started":1}
I20260619 14:54:35.270316 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:35.250356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35133 (local address 127.0.241.1:47317)
0619 14:54:35.251063 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:35.251103 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:35.252784 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:35.253521 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:35.253533 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:35.254319 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:35.255139 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:35.255167 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:35.256549 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:35.256556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:35.258677 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:35.258696 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:35.259479 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:35.259490 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:35.259839 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:35.260524 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:35.260556 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:35.263103 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:35.266690 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:35.266699 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:35.266712 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:35.267099 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:35.267475 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:35.267483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:35.267487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:35.267671 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:35.268159 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:35.268166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:35.268487 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 14:54:35.268742 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 14:54:35.269227 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":280,"threads_started":1}
I20260619 14:54:35.271858 20464 heartbeater.cc:347] Connected to a master server at 127.0.241.62:35133
I20260619 14:54:35.272185 20464 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:35.273106 20464 heartbeater.cc:511] Master 127.0.241.62:35133 requested a full tablet report, sending...
I20260619 14:54:35.275462 20274 ts_manager.cc:194] Registered new tserver with Master: 3ffdbc32cb864ee6827d1a6bc0ed61ff (127.0.241.1:45377)
I20260619 14:54:35.277657 20274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.1@KRBTEST.COM'} at 127.0.241.1:47317
WARNING: no policy specified for kudu/127.0.241.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
Entry for principal kudu/127.0.241.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab.
I20260619 14:54:35.310106 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35133
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:43891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:35.628324 20470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:35.628583 20470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:35.628739 20470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:35.636186 20470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:35.636291 20470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:35.636459 20470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:54:35.646027 20470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:43891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.2.keytab
--principal=kudu/127.0.241.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:35.647225 20470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:35.648593 20470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:35.661121 20475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:35.661211 20478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:35.661223 20476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:35.662002 20470 server_base.cc:1034] running on GCE node
I20260619 14:54:35.662576 20470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:35.663578 20470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:35.664783 20470 hybrid_clock.cc:648] HybridClock initialized: now 1781880875664734 us; error 38 us; skew 500 ppm
I20260619 14:54:35.665089 20470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:35 dist-test-slave-p3ml krb5kdc[20229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:35.671804 20470 init.cc:377] Logged in from keytab as kudu/127.0.241.2@KRBTEST.COM (short username kudu)
I20260619 14:54:35.673247 20470 webserver.cc:460] Webserver started at http://127.0.241.2:33973/ using document root <none> and password file <none>
I20260619 14:54:35.673940 20470 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:35.674036 20470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:35.674285 20470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:35.677034 20470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "d23fd822f4c8478182ffcb38937807e3"
format_stamp: "Formatted at 2026-06-19 14:54:35 on dist-test-slave-p3ml"
server_key: "f012a0f0562538562f096c280085311a"
server_key_iv: "e49ac57fc656c347ca47493ab09e5a7f"
server_key_version: "encryptionkey@0"
I20260619 14:54:35.677737 20470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "d23fd822f4c8478182ffcb38937807e3"
format_stamp: "Formatted at 2026-06-19 14:54:35 on dist-test-slave-p3ml"
server_key: "f012a0f0562538562f096c280085311a"
server_key_iv: "e49ac57fc656c347ca47493ab09e5a7f"
server_key_version: "encryptionkey@0"
I20260619 14:54:35.682394 20470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260619 14:54:35.685664 20485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:35.687165 20470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:54:35.687326 20470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "d23fd822f4c8478182ffcb38937807e3"
format_stamp: "Formatted at 2026-06-19 14:54:35 on dist-test-slave-p3ml"
server_key: "f012a0f0562538562f096c280085311a"
server_key_iv: "e49ac57fc656c347ca47493ab09e5a7f"
server_key_version: "encryptionkey@0"
I20260619 14:54:35.687562 20470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:35.707551 20470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:35.711243 20470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:35.711572 20470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:35.712450 20470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:35.713691 20470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:35.713762 20470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:35.713922 20470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:35.713973 20470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:35.751821 20470 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:39511
I20260619 14:54:35.751861 20597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:39511 every 8 connection(s)
I20260619 14:54:35.753074 20470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:54:35.760295 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 20470
I20260619 14:54:35.760481 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:54:35.761123 964 external_mini_cluster.cc:1428] Setting key da388ada7c0f127c052346022aaf1b30
Jun 19 14:54:35 dist-test-slave-p3ml krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.2@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:54:35.775564 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:35.755551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:55523 (local address 127.0.241.62:35133)
0619 14:54:35.755723 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:35.755727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:35.757003 (+ 1276us) server_negotiation.cc:407] Connection header received
0619 14:54:35.758837 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:35.758843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:35.758938 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:35.759047 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:35.760662 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:35.761854 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:35.763949 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:35.764743 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:35.764945 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:35.768613 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:35.768622 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:35.768631 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:35.768680 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:35.772246 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:35.773005 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:35.773012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:35.773016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:35.773172 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:35.773639 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:35.773644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:35.773648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:35.773961 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:35.774114 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:35.774900 (+ 786us) server_negotiation.cc:299] Negotiation successful
0619 14:54:35.775135 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:54:35.776566 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:35.756135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35133 (local address 127.0.241.2:55523)
0619 14:54:35.756839 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:35.756884 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:35.758563 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:35.759283 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:35.759291 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:35.759867 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:35.760460 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:35.760478 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:35.762036 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:35.762041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:35.763811 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:35.763823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:35.764994 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:35.764999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:35.765213 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:35.765904 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:35.765930 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:35.768341 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:35.772398 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:35.772409 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:35.772425 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:35.772866 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:35.773298 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:35.773303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:35.773308 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:35.773506 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:35.774186 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:35.774192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:35.774503 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 14:54:35.774783 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 14:54:35.775336 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":268,"threads_started":1}
I20260619 14:54:35.778052 20598 heartbeater.cc:347] Connected to a master server at 127.0.241.62:35133
I20260619 14:54:35.778395 20598 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:35.779150 20598 heartbeater.cc:511] Master 127.0.241.62:35133 requested a full tablet report, sending...
I20260619 14:54:35.780706 20274 ts_manager.cc:194] Registered new tserver with Master: d23fd822f4c8478182ffcb38937807e3 (127.0.241.2:39511)
I20260619 14:54:35.781946 20274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.2@KRBTEST.COM'} at 127.0.241.2:55523
WARNING: no policy specified for kudu/127.0.241.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
Entry for principal kudu/127.0.241.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab.
I20260619 14:54:35.815599 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35133
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:43891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:36.135072 20603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:36.135331 20603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:36.135535 20603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:36.142668 20603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:36.142795 20603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:36.142970 20603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:54:36.152421 20603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:43891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/krb5kdc/kudu_127.0.241.3.keytab
--principal=kudu/127.0.241.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:35133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:36.153580 20603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:36.154886 20603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:36.167806 20608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:36.167853 20609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:36.167889 20611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:36.168641 20603 server_base.cc:1034] running on GCE node
I20260619 14:54:36.169186 20603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:36.170027 20603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:36.171265 20603 hybrid_clock.cc:648] HybridClock initialized: now 1781880876171189 us; error 57 us; skew 500 ppm
I20260619 14:54:36.171591 20603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:36 dist-test-slave-p3ml krb5kdc[20229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:36.174571 20603 init.cc:377] Logged in from keytab as kudu/127.0.241.3@KRBTEST.COM (short username kudu)
I20260619 14:54:36.176019 20603 webserver.cc:460] Webserver started at http://127.0.241.3:43625/ using document root <none> and password file <none>
I20260619 14:54:36.176671 20603 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:36.176811 20603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:36.177104 20603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:36.179971 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "9aa5cfb5105c432c88df81018474c697"
format_stamp: "Formatted at 2026-06-19 14:54:36 on dist-test-slave-p3ml"
server_key: "c6a5da8d173dd9d368360bb4e4dc1676"
server_key_iv: "2bb31ff8b0601b4b37f5b39114794f8a"
server_key_version: "encryptionkey@0"
I20260619 14:54:36.180857 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "9aa5cfb5105c432c88df81018474c697"
format_stamp: "Formatted at 2026-06-19 14:54:36 on dist-test-slave-p3ml"
server_key: "c6a5da8d173dd9d368360bb4e4dc1676"
server_key_iv: "2bb31ff8b0601b4b37f5b39114794f8a"
server_key_version: "encryptionkey@0"
I20260619 14:54:36.185900 20603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:54:36.189249 20618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:36.190517 20603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:54:36.190876 20603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "9aa5cfb5105c432c88df81018474c697"
format_stamp: "Formatted at 2026-06-19 14:54:36 on dist-test-slave-p3ml"
server_key: "c6a5da8d173dd9d368360bb4e4dc1676"
server_key_iv: "2bb31ff8b0601b4b37f5b39114794f8a"
server_key_version: "encryptionkey@0"
I20260619 14:54:36.191059 20603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:36.216107 20603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:36.220331 20603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:36.220604 20603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:36.221551 20603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:36.222896 20603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:36.222982 20603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:36.223083 20603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:36.223133 20603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:36.256757 20603 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:46053
I20260619 14:54:36.256814 20730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:46053 every 8 connection(s)
I20260619 14:54:36.258055 20603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:54:36.267625 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 20603
I20260619 14:54:36.267807 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:54:36.268280 964 external_mini_cluster.cc:1428] Setting key ec8ff0a73d17f3f9421c219ecef63c5c
Jun 19 14:54:36 dist-test-slave-p3ml krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.3@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:54:36.277412 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.260583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:49775 (local address 127.0.241.62:35133)
0619 14:54:36.260764 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:36.260768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:36.261869 (+ 1101us) server_negotiation.cc:407] Connection header received
0619 14:54:36.263680 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:36.263688 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:36.263782 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:36.263893 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:36.265311 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.266069 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.267791 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.268271 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.268417 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.271928 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:36.271932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:36.271936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:36.271966 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:36.274473 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:36.275151 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:36.275156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:36.275160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:36.275288 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:36.275753 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:36.275758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:36.275761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:36.276039 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:36.276195 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:36.276831 (+ 636us) server_negotiation.cc:299] Negotiation successful
0619 14:54:36.277075 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:54:36.278267 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.261048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35133 (local address 127.0.241.3:49775)
0619 14:54:36.261717 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:36.261757 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:36.263382 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:36.264103 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:36.264110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:36.264595 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:36.265137 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:36.265154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.266207 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.266216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:36.267614 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:36.267627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.268536 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.268544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:36.268804 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.269500 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:36.269522 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:36.271752 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:36.274582 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:36.274588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:36.274598 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:36.275013 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:36.275407 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:36.275412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:36.275416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:36.275634 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:36.276165 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:36.276172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:36.276470 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 14:54:36.276717 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 14:54:36.277198 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":274,"threads_started":1}
I20260619 14:54:36.279521 20731 heartbeater.cc:347] Connected to a master server at 127.0.241.62:35133
I20260619 14:54:36.279806 20731 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:36.280448 20731 heartbeater.cc:511] Master 127.0.241.62:35133 requested a full tablet report, sending...
I20260619 14:54:36.281440 20464 heartbeater.cc:503] Master 127.0.241.62:35133 was elected leader, sending a full tablet report...
I20260619 14:54:36.282047 20274 ts_manager.cc:194] Registered new tserver with Master: 9aa5cfb5105c432c88df81018474c697 (127.0.241.3:46053)
I20260619 14:54:36.283210 20274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.241.3@KRBTEST.COM'} at 127.0.241.3:49775
I20260619 14:54:36.283389 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:54:36.301678 20466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.288253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.0.241.62:35133)
0619 14:54:36.288426 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:36.288430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:36.288940 (+ 510us) server_negotiation.cc:407] Connection header received
0619 14:54:36.289199 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:36.289203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:36.289261 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:36.289364 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:36.290824 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.291608 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.292996 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.293465 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.293619 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.296190 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:36.296195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:36.296198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:36.296229 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:36.298830 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:36.299594 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:36.299598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:36.299601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:36.299744 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:36.300150 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:36.300157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:36.300161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:36.300481 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:36.300768 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:36.301084 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 14:54:36.301311 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 14:54:36.305281 20274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:54:36.311933 20274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:54:36.337536 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.325388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.0.241.3:46053)
0619 14:54:36.326263 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:36.326275 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:36.328074 (+ 1799us) server_negotiation.cc:407] Connection header received
0619 14:54:36.328409 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:36.328422 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:36.328749 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:36.328951 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:36.330151 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.332176 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.335041 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.336375 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.336507 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.336760 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:36.336986 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 14:54:36.337155 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":463,"threads_started":1}
I20260619 14:54:36.337579 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.324667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:46053 (local address 127.0.0.1:52918)
0619 14:54:36.327945 (+ 3278us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:36.327966 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:36.328198 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:36.329404 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:36.329415 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:36.329457 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:36.329925 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:36.329940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.332318 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.332325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:36.334808 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:36.334836 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.336549 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.336555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:36.336748 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.336800 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 14:54:36.336932 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 14:54:36.337111 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3110,"mutex_wait_us":48,"thread_start_us":1033,"threads_started":1}
I20260619 14:54:36.338582 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.323242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:39511 (local address 127.0.0.1:50282)
0619 14:54:36.324070 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:36.324119 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:36.324328 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:36.331509 (+ 7181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:36.331513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:36.331537 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:36.332002 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:36.332013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.333497 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.333502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:36.335537 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:36.335552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.337339 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.337349 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:36.337555 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.337599 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 14:54:36.337676 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 14:54:36.337823 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":92,"thread_start_us":847,"threads_started":1}
I20260619 14:54:36.338707 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.323369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:45377 (local address 127.0.0.1:43530)
0619 14:54:36.325313 (+ 1944us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:36.325334 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:36.325464 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:36.331380 (+ 5916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:36.331384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:36.331409 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:36.331941 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:36.331953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.334054 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.334064 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:36.336149 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:36.336159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.337959 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.337966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:36.338125 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.338158 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:54:36.338226 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:54:36.338354 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1773,"mutex_wait_us":179,"thread_start_us":4246,"threads_started":1}
I20260619 14:54:36.338821 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.323297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50282 (local address 127.0.241.2:39511)
0619 14:54:36.330927 (+ 7630us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:36.330937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:36.330966 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:54:36.331059 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:36.331066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:36.331271 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:36.332202 (+ 931us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:36.332412 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.333365 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.335690 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.337108 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.338057 (+ 949us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.338251 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:36.338337 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 14:54:36.338498 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7460,"thread_start_us":7336,"threads_started":1}
I20260619 14:54:36.339800 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.323951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.0.241.1:45377)
0619 14:54:36.330720 (+ 6769us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:36.330729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:36.330754 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:54:36.330855 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:36.330867 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:36.331098 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:36.332118 (+ 1020us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:36.332350 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.333896 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.336291 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.337687 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.338942 (+ 1255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.339202 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:36.339289 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 14:54:36.339485 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6581,"thread_start_us":6350,"threads_started":1}
I20260619 14:54:36.342108 20666 tablet_service.cc:1467] Processing CreateTablet for tablet 42aabfd164944787a9526cceaf746c00 (DEFAULT_TABLE table=test-table [id=4c4ed573dd0b44209ec0f633046aeed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:36.342415 20399 tablet_service.cc:1467] Processing CreateTablet for tablet 42aabfd164944787a9526cceaf746c00 (DEFAULT_TABLE table=test-table [id=4c4ed573dd0b44209ec0f633046aeed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:36.342543 20533 tablet_service.cc:1467] Processing CreateTablet for tablet 42aabfd164944787a9526cceaf746c00 (DEFAULT_TABLE table=test-table [id=4c4ed573dd0b44209ec0f633046aeed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:36.343629 20399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42aabfd164944787a9526cceaf746c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:36.343660 20666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42aabfd164944787a9526cceaf746c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:36.344106 20533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42aabfd164944787a9526cceaf746c00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:36.354002 20749 tablet_bootstrap.cc:492] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff: Bootstrap starting.
I20260619 14:54:36.358009 20750 tablet_bootstrap.cc:492] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697: Bootstrap starting.
I20260619 14:54:36.358307 20749 tablet_bootstrap.cc:654] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:36.359699 20749 log.cc:824] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:36.359748 20751 tablet_bootstrap.cc:492] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3: Bootstrap starting.
I20260619 14:54:36.361778 20750 tablet_bootstrap.cc:654] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:36.362828 20751 tablet_bootstrap.cc:654] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:36.363241 20750 log.cc:824] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:36.363950 20751 log.cc:824] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:36.367193 20751 tablet_bootstrap.cc:492] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3: No bootstrap required, opened a new log
I20260619 14:54:36.367197 20749 tablet_bootstrap.cc:492] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff: No bootstrap required, opened a new log
I20260619 14:54:36.367504 20749 ts_tablet_manager.cc:1397] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20260619 14:54:36.367491 20751 ts_tablet_manager.cc:1397] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 14:54:36.367774 20750 tablet_bootstrap.cc:492] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697: No bootstrap required, opened a new log
I20260619 14:54:36.368031 20750 ts_tablet_manager.cc:1397] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260619 14:54:36.375196 20750 raft_consensus.cc:348] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23fd822f4c8478182ffcb38937807e3" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 39511 } } peers { permanent_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45377 } } peers { permanent_uuid: "9aa5cfb5105c432c88df81018474c697" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 46053 } }
I20260619 14:54:36.375684 20750 raft_consensus.cc:374] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:36.375787 20750 raft_consensus.cc:729] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa5cfb5105c432c88df81018474c697, State: Initialized, Role: FOLLOWER
I20260619 14:54:36.376438 20750 consensus_queue.cc:260] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23fd822f4c8478182ffcb38937807e3" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 39511 } } peers { permanent_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45377 } } peers { permanent_uuid: "9aa5cfb5105c432c88df81018474c697" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 46053 } }
I20260619 14:54:36.377584 20731 heartbeater.cc:503] Master 127.0.241.62:35133 was elected leader, sending a full tablet report...
I20260619 14:54:36.378273 20750 ts_tablet_manager.cc:1428] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20260619 14:54:36.378227 20749 raft_consensus.cc:348] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23fd822f4c8478182ffcb38937807e3" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 39511 } } peers { permanent_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45377 } } peers { permanent_uuid: "9aa5cfb5105c432c88df81018474c697" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 46053 } }
I20260619 14:54:36.378224 20751 raft_consensus.cc:348] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23fd822f4c8478182ffcb38937807e3" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 39511 } } peers { permanent_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45377 } } peers { permanent_uuid: "9aa5cfb5105c432c88df81018474c697" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 46053 } }
I20260619 14:54:36.378618 20749 raft_consensus.cc:374] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:36.378620 20751 raft_consensus.cc:374] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:36.378713 20749 raft_consensus.cc:729] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffdbc32cb864ee6827d1a6bc0ed61ff, State: Initialized, Role: FOLLOWER
I20260619 14:54:36.378707 20751 raft_consensus.cc:729] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23fd822f4c8478182ffcb38937807e3, State: Initialized, Role: FOLLOWER
I20260619 14:54:36.379264 20751 consensus_queue.cc:260] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23fd822f4c8478182ffcb38937807e3" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 39511 } } peers { permanent_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45377 } } peers { permanent_uuid: "9aa5cfb5105c432c88df81018474c697" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 46053 } }
I20260619 14:54:36.379266 20749 consensus_queue.cc:260] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23fd822f4c8478182ffcb38937807e3" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 39511 } } peers { permanent_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45377 } } peers { permanent_uuid: "9aa5cfb5105c432c88df81018474c697" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 46053 } }
I20260619 14:54:36.381361 20749 ts_tablet_manager.cc:1428] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260619 14:54:36.381501 20598 heartbeater.cc:503] Master 127.0.241.62:35133 was elected leader, sending a full tablet report...
I20260619 14:54:36.382194 20751 ts_tablet_manager.cc:1428] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
I20260619 14:54:36.394594 20756 raft_consensus.cc:482] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:36.394871 20756 raft_consensus.cc:504] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23fd822f4c8478182ffcb38937807e3" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 39511 } } peers { permanent_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45377 } } peers { permanent_uuid: "9aa5cfb5105c432c88df81018474c697" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 46053 } }
I20260619 14:54:36.396540 20756 leader_election.cc:288] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d23fd822f4c8478182ffcb38937807e3 (127.0.241.2:39511), 9aa5cfb5105c432c88df81018474c697 (127.0.241.3:46053)
I20260619 14:54:36.402940 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.396869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:53071 (local address 127.0.241.2:39511)
0619 14:54:36.397166 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:36.397174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:36.397907 (+ 733us) server_negotiation.cc:407] Connection header received
0619 14:54:36.398109 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:36.398113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:36.398167 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:36.398256 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:36.398839 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.399594 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.401275 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.402034 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.402180 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.402280 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:36.402551 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 14:54:36.402672 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260619 14:54:36.403239 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.397050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:52085 (local address 127.0.241.3:46053)
0619 14:54:36.397257 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:36.397265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:36.398082 (+ 817us) server_negotiation.cc:407] Connection header received
0619 14:54:36.398266 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:36.398275 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:36.398360 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:36.398458 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:36.399123 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.399910 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.401593 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.402346 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.402490 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.402595 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:36.402878 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 14:54:36.402998 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:54:36.403267 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.396985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:39511 (local address 127.0.241.1:53071)
0619 14:54:36.397802 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:36.397818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:36.397980 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:36.398271 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:36.398305 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:36.398329 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:36.398705 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:36.398720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.399708 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.399716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:36.401133 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:36.401144 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.402245 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.402251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:36.402400 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.402423 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:54:36.402528 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 14:54:36.402657 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":1,"thread_start_us":436,"threads_started":1}
I20260619 14:54:36.403474 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.396989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:46053 (local address 127.0.241.1:52085)
0619 14:54:36.397985 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:36.397999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:36.398144 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:36.398475 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:36.398479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:36.398508 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:36.398954 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:36.398966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.400044 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.400050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:36.401448 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:36.401459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.402558 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.402564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:36.402724 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.402746 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:54:36.402852 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:54:36.402977 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":49,"thread_start_us":576,"threads_started":1}
I20260619 14:54:36.404328 20553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42aabfd164944787a9526cceaf746c00" candidate_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d23fd822f4c8478182ffcb38937807e3" is_pre_election: true
I20260619 14:54:36.404403 20686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42aabfd164944787a9526cceaf746c00" candidate_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa5cfb5105c432c88df81018474c697" is_pre_election: true
I20260619 14:54:36.404902 20686 raft_consensus.cc:2454] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ffdbc32cb864ee6827d1a6bc0ed61ff in term 0.
I20260619 14:54:36.404898 20553 raft_consensus.cc:2454] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ffdbc32cb864ee6827d1a6bc0ed61ff in term 0.
I20260619 14:54:36.405529 20353 leader_election.cc:302] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ffdbc32cb864ee6827d1a6bc0ed61ff, 9aa5cfb5105c432c88df81018474c697; no voters:
I20260619 14:54:36.405903 20756 raft_consensus.cc:2781] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:54:36.406061 20756 raft_consensus.cc:482] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:54:36.406169 20756 raft_consensus.cc:3037] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:36.408396 20756 raft_consensus.cc:504] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23fd822f4c8478182ffcb38937807e3" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 39511 } } peers { permanent_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45377 } } peers { permanent_uuid: "9aa5cfb5105c432c88df81018474c697" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 46053 } }
I20260619 14:54:36.408957 20756 leader_election.cc:288] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [CANDIDATE]: Term 1 election: Requested vote from peers d23fd822f4c8478182ffcb38937807e3 (127.0.241.2:39511), 9aa5cfb5105c432c88df81018474c697 (127.0.241.3:46053)
I20260619 14:54:36.409345 20553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42aabfd164944787a9526cceaf746c00" candidate_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d23fd822f4c8478182ffcb38937807e3"
I20260619 14:54:36.409416 20686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42aabfd164944787a9526cceaf746c00" candidate_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa5cfb5105c432c88df81018474c697"
I20260619 14:54:36.409557 20553 raft_consensus.cc:3037] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:36.409587 20686 raft_consensus.cc:3037] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:36.412814 20686 raft_consensus.cc:2454] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ffdbc32cb864ee6827d1a6bc0ed61ff in term 1.
I20260619 14:54:36.412825 20553 raft_consensus.cc:2454] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ffdbc32cb864ee6827d1a6bc0ed61ff in term 1.
I20260619 14:54:36.413231 20352 leader_election.cc:302] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ffdbc32cb864ee6827d1a6bc0ed61ff, d23fd822f4c8478182ffcb38937807e3; no voters:
I20260619 14:54:36.413532 20756 raft_consensus.cc:2781] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:36.413985 20756 raft_consensus.cc:686] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 1 LEADER]: Becoming Leader. State: Replica: 3ffdbc32cb864ee6827d1a6bc0ed61ff, State: Running, Role: LEADER
I20260619 14:54:36.414410 20756 consensus_queue.cc:237] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23fd822f4c8478182ffcb38937807e3" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 39511 } } peers { permanent_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45377 } } peers { permanent_uuid: "9aa5cfb5105c432c88df81018474c697" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 46053 } }
I20260619 14:54:36.418505 20272 catalog_manager.cc:5495] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ffdbc32cb864ee6827d1a6bc0ed61ff (127.0.241.1). New cstate: current_term: 1 leader_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d23fd822f4c8478182ffcb38937807e3" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 39511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ffdbc32cb864ee6827d1a6bc0ed61ff" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 45377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9aa5cfb5105c432c88df81018474c697" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 46053 } health_report { overall_health: UNKNOWN } } }
I20260619 14:54:36.434684 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.430034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.0.241.1:45377)
0619 14:54:36.430200 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:36.430204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:36.430221 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:54:36.430328 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:36.430332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:36.430425 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:36.430547 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:36.430854 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.431600 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.432908 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.433280 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.433367 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.433764 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:36.434262 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:54:36.434367 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:36.434432 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 14:54:36.434494 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:54:36.442165 20686 raft_consensus.cc:1261] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697 [term 1 FOLLOWER]: Refusing update from remote peer 3ffdbc32cb864ee6827d1a6bc0ed61ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:36.442183 20553 raft_consensus.cc:1261] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3 [term 1 FOLLOWER]: Refusing update from remote peer 3ffdbc32cb864ee6827d1a6bc0ed61ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:36.443077 20760 consensus_queue.cc:1035] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa5cfb5105c432c88df81018474c697" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 46053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:36.443411 20756 consensus_queue.cc:1035] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23fd822f4c8478182ffcb38937807e3" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 39511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:36.454685 20763 mvcc.cc:205] Tried to move back new op lower bound from 7298584069898379264 to 7298584069796888576. Current Snapshot: MvccSnapshot[applied={T|T < 7298584069898379264}]
I20260619 14:54:36.455718 20764 mvcc.cc:205] Tried to move back new op lower bound from 7298584069898379264 to 7298584069796888576. Current Snapshot: MvccSnapshot[applied={T|T < 7298584069898379264}]
I20260619 14:54:36.468080 20272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:36.468408 20272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:36.472069 20272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34a1b2217f864c7b9b459fe517d8f08a: Sending DeleteTablet for 3 replicas of tablet 42aabfd164944787a9526cceaf746c00
I20260619 14:54:36.473246 20399 tablet_service.cc:1514] Processing DeleteTablet for tablet 42aabfd164944787a9526cceaf746c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:36 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:43530
I20260619 14:54:36.473405 20533 tablet_service.cc:1514] Processing DeleteTablet for tablet 42aabfd164944787a9526cceaf746c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:36 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:50282
I20260619 14:54:36.473490 20666 tablet_service.cc:1514] Processing DeleteTablet for tablet 42aabfd164944787a9526cceaf746c00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:36 UTC) from {username='kudu', principal='kudu/127.0.241.62@KRBTEST.COM'} at 127.0.0.1:52918
I20260619 14:54:36.475646 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 20336
I20260619 14:54:36.486145 20773 tablet_replica.cc:330] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff: stopping tablet replica
I20260619 14:54:36.486677 20773 raft_consensus.cc:2229] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:54:36.487565 20773 raft_consensus.cc:2258] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:36.490031 20773 ts_tablet_manager.cc:1905] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:36.499590 20773 ts_tablet_manager.cc:1918] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:36.499800 20773 log.cc:1196] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-0/wal/wals/42aabfd164944787a9526cceaf746c00
I20260619 14:54:36.500413 20773 ts_tablet_manager.cc:1939] T 42aabfd164944787a9526cceaf746c00 P 3ffdbc32cb864ee6827d1a6bc0ed61ff: Deleting consensus metadata
I20260619 14:54:36.502460 20260 catalog_manager.cc:4841] TS 3ffdbc32cb864ee6827d1a6bc0ed61ff (127.0.241.1:45377): tablet 42aabfd164944787a9526cceaf746c00 (table test-table [id=4c4ed573dd0b44209ec0f633046aeed3]) successfully deleted
I20260619 14:54:36.503089 20775 tablet_replica.cc:330] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697: stopping tablet replica
I20260619 14:54:36.503531 20775 raft_consensus.cc:2229] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:36.504084 20775 raft_consensus.cc:2258] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:36.506042 20775 ts_tablet_manager.cc:1905] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:36.507776 20774 tablet_replica.cc:330] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3: stopping tablet replica
I20260619 14:54:36.508183 20774 raft_consensus.cc:2229] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:36.508656 20774 raft_consensus.cc:2258] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:36.511183 20774 ts_tablet_manager.cc:1905] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:36.512646 20775 ts_tablet_manager.cc:1918] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:36.512815 20775 log.cc:1196] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/42aabfd164944787a9526cceaf746c00
I20260619 14:54:36.513311 20775 ts_tablet_manager.cc:1939] T 42aabfd164944787a9526cceaf746c00 P 9aa5cfb5105c432c88df81018474c697: Deleting consensus metadata
Jun 19 14:54:36 dist-test-slave-p3ml krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.1@KRBTEST.COM
I20260619 14:54:36.514981 20259 catalog_manager.cc:4841] TS 9aa5cfb5105c432c88df81018474c697 (127.0.241.3:46053): tablet 42aabfd164944787a9526cceaf746c00 (table test-table [id=4c4ed573dd0b44209ec0f633046aeed3]) successfully deleted
I20260619 14:54:36.517422 20774 ts_tablet_manager.cc:1918] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:36.517583 20774 log.cc:1196] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/42aabfd164944787a9526cceaf746c00
I20260619 14:54:36.518023 20774 ts_tablet_manager.cc:1939] T 42aabfd164944787a9526cceaf746c00 P d23fd822f4c8478182ffcb38937807e3: Deleting consensus metadata
I20260619 14:54:36.519364 20258 catalog_manager.cc:4841] TS d23fd822f4c8478182ffcb38937807e3 (127.0.241.2:39511): tablet 42aabfd164944787a9526cceaf746c00 (table test-table [id=4c4ed573dd0b44209ec0f633046aeed3]) successfully deleted
I20260619 14:54:36.521991 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.476990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.0.241.1:45377)
0619 14:54:36.477207 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:36.477216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:36.501019 (+ 23803us) server_negotiation.cc:407] Connection header received
0619 14:54:36.501508 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:36.501514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:36.501607 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:36.501681 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:36.504822 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.506024 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.508033 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.508726 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.508819 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.514743 (+ 5924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:36.514758 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:36.514775 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:36.514833 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:36.518891 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:36.519824 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:36.519833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:36.519837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:36.520015 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:36.520399 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:36.520405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:36.520408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:36.520862 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:36.521154 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:36.521373 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 14:54:36.521608 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:54:36.522758 20459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:36.620179 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 20336
I20260619 14:54:36.638837 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 20470
Jun 19 14:54:36 dist-test-slave-p3ml krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.2@KRBTEST.COM
I20260619 14:54:36.653177 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.639597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.0.241.2:39511)
0619 14:54:36.639755 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:36.639760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:36.639778 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:54:36.639915 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:36.639919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:36.639979 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:36.640060 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:36.641278 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.642008 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.643248 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.643711 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.643809 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.647929 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:36.647941 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:36.647956 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:36.647990 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:36.650570 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:36.651170 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:36.651176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:36.651178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:36.651292 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:36.651929 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:36.651932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:36.651934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:36.652326 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:36.652537 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:36.652769 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 14:54:36.652933 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:54:36.653801 20593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:36.741243 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 20470
I20260619 14:54:36.757800 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 20603
Jun 19 14:54:36 dist-test-slave-p3ml krb5kdc[20229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.3@KRBTEST.COM
I20260619 14:54:36.773000 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:36.758532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.0.241.3:46053)
0619 14:54:36.758720 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:36.758728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:36.758748 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:54:36.758953 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:36.758958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:36.759024 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:36.759138 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:36.760516 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.761253 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.762557 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:36.762955 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:36.763112 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:36.767608 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:36.767623 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:36.767639 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:36.767671 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:36.770179 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:36.770866 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:36.770872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:36.770875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:36.771000 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:36.771377 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:36.771384 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:36.771387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:36.771860 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:36.772153 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:36.772446 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 14:54:36.772668 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:54:36.773653 20726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:36.861996 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 20603
I20260619 14:54:36.879045 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 20242
I20260619 14:54:36.880110 20304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:36.990103 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 20242
2026-06-19T14:54:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4144 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T14:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:37Z Disabled control of system clock
I20260619 14:54:37.066469 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45227
--webserver_interface=127.0.241.62
--webserver_port=0
--builtin_ntp_servers=127.0.241.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:45227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:54:37.397811 20784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:37.398087 20784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:37.398175 20784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:37.405166 20784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:37.405267 20784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:37.405304 20784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:37.405364 20784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:37.405408 20784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:37.414211 20784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:45227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:37.415393 20784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:37.417075 20784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:37.428615 20790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:37.428737 20789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:37.428646 20792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:37.429435 20784 server_base.cc:1034] running on GCE node
I20260619 14:54:37.430128 20784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:37.431360 20784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:37.432585 20784 hybrid_clock.cc:648] HybridClock initialized: now 1781880877432529 us; error 50 us; skew 500 ppm
I20260619 14:54:37.432888 20784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:54:37.434970 20784 webserver.cc:460] Webserver started at http://127.0.241.62:40251/ using document root <none> and password file <none>
I20260619 14:54:37.435631 20784 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:37.435719 20784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:37.435979 20784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:37.438632 20784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "a340ffb4b9434d528cc6737258a34c0f"
format_stamp: "Formatted at 2026-06-19 14:54:37 on dist-test-slave-p3ml"
server_key: "96ea98ce290958ea1180c49c5d34756f"
server_key_iv: "919728fea7c73c88672b8981ce411da0"
server_key_version: "encryptionkey@0"
I20260619 14:54:37.439333 20784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "a340ffb4b9434d528cc6737258a34c0f"
format_stamp: "Formatted at 2026-06-19 14:54:37 on dist-test-slave-p3ml"
server_key: "96ea98ce290958ea1180c49c5d34756f"
server_key_iv: "919728fea7c73c88672b8981ce411da0"
server_key_version: "encryptionkey@0"
I20260619 14:54:37.444188 20784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:54:37.447274 20798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:37.448508 20784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 14:54:37.448693 20784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "a340ffb4b9434d528cc6737258a34c0f"
format_stamp: "Formatted at 2026-06-19 14:54:37 on dist-test-slave-p3ml"
server_key: "96ea98ce290958ea1180c49c5d34756f"
server_key_iv: "919728fea7c73c88672b8981ce411da0"
server_key_version: "encryptionkey@0"
I20260619 14:54:37.448905 20784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:37.492573 20784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:37.493489 20784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:37.493785 20784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:37.512465 20784 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:45227
I20260619 14:54:37.512485 20849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:45227 every 8 connection(s)
I20260619 14:54:37.513819 20784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:37.517349 20850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:37.519325 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 20784
I20260619 14:54:37.519527 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:37.519959 964 external_mini_cluster.cc:1428] Setting key bcc0b2e4032372c03baaeeb6771e5f45
I20260619 14:54:37.530438 20850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f: Bootstrap starting.
I20260619 14:54:37.534380 20850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:37.535385 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:37.521844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.0.241.62:45227)
0619 14:54:37.522905 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:37.522929 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:37.522985 (+ 56us) server_negotiation.cc:407] Connection header received
0619 14:54:37.525325 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:37.525351 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:37.525897 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:37.526324 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:37.526860 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:37.528943 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:37.530342 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:37.531408 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:37.531611 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:37.532711 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:37.532721 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:37.532738 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:37.533408 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:37.533515 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:37.533543 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:37.533646 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:37.533830 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 14:54:37.534132 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":324,"threads_started":1}
I20260619 14:54:37.535724 20850 log.cc:824] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:37.538555 20850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f: No bootstrap required, opened a new log
I20260619 14:54:37.546089 20850 raft_consensus.cc:348] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a340ffb4b9434d528cc6737258a34c0f" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45227 } }
I20260619 14:54:37.546432 20850 raft_consensus.cc:374] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:37.546550 20850 raft_consensus.cc:729] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a340ffb4b9434d528cc6737258a34c0f, State: Initialized, Role: FOLLOWER
I20260619 14:54:37.547075 20850 consensus_queue.cc:260] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a340ffb4b9434d528cc6737258a34c0f" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45227 } }
I20260619 14:54:37.547307 20850 raft_consensus.cc:388] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:37.547456 20850 raft_consensus.cc:482] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:37.547631 20850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:37.549721 20850 raft_consensus.cc:504] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a340ffb4b9434d528cc6737258a34c0f" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45227 } }
I20260619 14:54:37.550217 20850 leader_election.cc:302] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a340ffb4b9434d528cc6737258a34c0f; no voters:
I20260619 14:54:37.550763 20850 leader_election.cc:288] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:37.550908 20855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:37.551653 20855 raft_consensus.cc:686] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [term 1 LEADER]: Becoming Leader. State: Replica: a340ffb4b9434d528cc6737258a34c0f, State: Running, Role: LEADER
I20260619 14:54:37.552210 20855 consensus_queue.cc:237] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a340ffb4b9434d528cc6737258a34c0f" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45227 } }
I20260619 14:54:37.552875 20850 sys_catalog.cc:564] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:37.555392 20857 sys_catalog.cc:455] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a340ffb4b9434d528cc6737258a34c0f. Latest consensus state: current_term: 1 leader_uuid: "a340ffb4b9434d528cc6737258a34c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a340ffb4b9434d528cc6737258a34c0f" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45227 } } }
I20260619 14:54:37.555415 20856 sys_catalog.cc:455] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a340ffb4b9434d528cc6737258a34c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a340ffb4b9434d528cc6737258a34c0f" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45227 } } }
I20260619 14:54:37.555932 20857 sys_catalog.cc:458] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:37.555929 20856 sys_catalog.cc:458] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:37.557260 20862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:37.562179 20862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:37.574086 20862 catalog_manager.cc:1347] Generated new cluster ID: a96767bdb7c74f7785fc05bad5efa245
I20260619 14:54:37.574200 20862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:37.594393 20862 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:37.595722 20862 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:37.605468 20862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f: Generated new TSK 0
I20260619 14:54:37.606231 20862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:37.617154 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.1:0
--local_ip_for_outbound_sockets=127.0.241.1
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45227
--builtin_ntp_servers=127.0.241.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:54:37.936530 20874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:37.936846 20874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:37.936996 20874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:37.943997 20874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:37.944101 20874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:37.944267 20874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.1
I20260619 14:54:37.953399 20874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.241.1
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.1
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:37.954614 20874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:37.956390 20874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:37.969406 20880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:37.969442 20882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:37.969498 20879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:37.970415 20874 server_base.cc:1034] running on GCE node
I20260619 14:54:37.971014 20874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:37.971997 20874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:37.973223 20874 hybrid_clock.cc:648] HybridClock initialized: now 1781880877973180 us; error 40 us; skew 500 ppm
I20260619 14:54:37.973493 20874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:54:37.975667 20874 webserver.cc:460] Webserver started at http://127.0.241.1:45207/ using document root <none> and password file <none>
I20260619 14:54:37.976296 20874 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:37.976384 20874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:37.976681 20874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:37.979686 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/data/instance:
uuid: "2a4e7c79039c4b0eaed120a55a7abd76"
format_stamp: "Formatted at 2026-06-19 14:54:37 on dist-test-slave-p3ml"
server_key: "8985671f5f6bc2b4997e4aa898e802d4"
server_key_iv: "e40f43fc8fe0b542dcfd17fc78014175"
server_key_version: "encryptionkey@0"
I20260619 14:54:37.980384 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/wal/instance:
uuid: "2a4e7c79039c4b0eaed120a55a7abd76"
format_stamp: "Formatted at 2026-06-19 14:54:37 on dist-test-slave-p3ml"
server_key: "8985671f5f6bc2b4997e4aa898e802d4"
server_key_iv: "e40f43fc8fe0b542dcfd17fc78014175"
server_key_version: "encryptionkey@0"
I20260619 14:54:37.985064 20874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:54:37.988296 20888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:37.989552 20874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:54:37.989672 20874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/wal
uuid: "2a4e7c79039c4b0eaed120a55a7abd76"
format_stamp: "Formatted at 2026-06-19 14:54:37 on dist-test-slave-p3ml"
server_key: "8985671f5f6bc2b4997e4aa898e802d4"
server_key_iv: "e40f43fc8fe0b542dcfd17fc78014175"
server_key_version: "encryptionkey@0"
I20260619 14:54:37.989845 20874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:38.013264 20874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:38.014046 20874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:38.014315 20874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:38.015293 20874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:38.016671 20874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:38.016777 20874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:38.016906 20874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:38.017009 20874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:38.049322 20874 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.1:44171
I20260619 14:54:38.049418 21000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.1:44171 every 8 connection(s)
I20260619 14:54:38.050583 20874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/data/info.pb
I20260619 14:54:38.053634 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 20874
I20260619 14:54:38.053798 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-0/wal/instance
I20260619 14:54:38.054406 964 external_mini_cluster.cc:1428] Setting key a3af4d357541e89eb3546082b2c228fe
I20260619 14:54:38.057382 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.2:0
--local_ip_for_outbound_sockets=127.0.241.2
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45227
--builtin_ntp_servers=127.0.241.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:54:38.064236 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:38.053122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.1:47911 (local address 127.0.241.62:45227)
0619 14:54:38.055756 (+ 2634us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:38.055761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:38.055788 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:54:38.057679 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:38.057687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:38.057769 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:38.057896 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:38.058958 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.059775 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.061339 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.061764 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.061917 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:38.063301 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:38.063307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:38.063311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:38.063337 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:38.063385 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:38.063397 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:38.063526 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:38.063746 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 14:54:38.063857 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2529,"thread_start_us":2424,"threads_started":1}
I20260619 14:54:38.065258 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:38.053837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45227 (local address 127.0.241.1:47911)
0619 14:54:38.055230 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:38.055299 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:38.057333 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:38.058128 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:38.058136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:38.058232 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:38.058797 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:38.058813 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.059933 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.059938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:38.061183 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:38.061196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.062020 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.062030 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:38.062271 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:38.062984 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:54:38.062987 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:38.063027 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:54:38.063111 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:38.063528 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:38.063542 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:38.063552 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:54:38.063735 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 14:54:38.064142 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":411,"threads_started":1}
I20260619 14:54:38.066810 21001 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45227
I20260619 14:54:38.067080 21001 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:38.067823 21001 heartbeater.cc:511] Master 127.0.241.62:45227 requested a full tablet report, sending...
I20260619 14:54:38.070015 20815 ts_manager.cc:194] Registered new tserver with Master: 2a4e7c79039c4b0eaed120a55a7abd76 (127.0.241.1:44171)
I20260619 14:54:38.072268 20815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.241.1:47911
W20260619 14:54:38.388218 21005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:38.388482 21005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:38.388572 21005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:38.396152 21005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:38.396250 21005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:38.396360 21005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.2
I20260619 14:54:38.405177 21005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.241.2
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.2
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:38.406284 21005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:38.407908 21005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:38.420413 21010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:38.420531 21011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:38.420555 21013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:38.421550 21005 server_base.cc:1034] running on GCE node
I20260619 14:54:38.422133 21005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:38.423033 21005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:38.424221 21005 hybrid_clock.cc:648] HybridClock initialized: now 1781880878424188 us; error 32 us; skew 500 ppm
I20260619 14:54:38.424511 21005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:54:38.426565 21005 webserver.cc:460] Webserver started at http://127.0.241.2:45663/ using document root <none> and password file <none>
I20260619 14:54:38.427201 21005 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:38.427304 21005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:38.427664 21005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:38.430321 21005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/data/instance:
uuid: "dfc926eab2ca4d5d8c90c39f629a447a"
format_stamp: "Formatted at 2026-06-19 14:54:38 on dist-test-slave-p3ml"
server_key: "85fb522fe82c27b99866527fdcfa73a2"
server_key_iv: "b93005b837714b81894d68bb538e0c0a"
server_key_version: "encryptionkey@0"
I20260619 14:54:38.431062 21005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/wal/instance:
uuid: "dfc926eab2ca4d5d8c90c39f629a447a"
format_stamp: "Formatted at 2026-06-19 14:54:38 on dist-test-slave-p3ml"
server_key: "85fb522fe82c27b99866527fdcfa73a2"
server_key_iv: "b93005b837714b81894d68bb538e0c0a"
server_key_version: "encryptionkey@0"
I20260619 14:54:38.435899 21005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 14:54:38.439002 21019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:38.440249 21005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:54:38.440402 21005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/wal
uuid: "dfc926eab2ca4d5d8c90c39f629a447a"
format_stamp: "Formatted at 2026-06-19 14:54:38 on dist-test-slave-p3ml"
server_key: "85fb522fe82c27b99866527fdcfa73a2"
server_key_iv: "b93005b837714b81894d68bb538e0c0a"
server_key_version: "encryptionkey@0"
I20260619 14:54:38.440599 21005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:38.482467 21005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:38.483402 21005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:38.483706 21005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:38.484714 21005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:38.486166 21005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:38.486284 21005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:38.486404 21005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:38.486487 21005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:38.519546 21005 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.2:35323
I20260619 14:54:38.519585 21131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.2:35323 every 8 connection(s)
I20260619 14:54:38.520861 21005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/data/info.pb
I20260619 14:54:38.524456 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 21005
I20260619 14:54:38.524636 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/wal/instance
I20260619 14:54:38.525170 964 external_mini_cluster.cc:1428] Setting key afd17805c2060d93b24c7855f6d05988
I20260619 14:54:38.528568 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.241.3:0
--local_ip_for_outbound_sockets=127.0.241.3
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45227
--builtin_ntp_servers=127.0.241.20:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:54:38.535844 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:38.523520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.2:38163 (local address 127.0.241.62:45227)
0619 14:54:38.523746 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:38.523752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:38.525173 (+ 1421us) server_negotiation.cc:407] Connection header received
0619 14:54:38.527370 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:38.527376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:38.527476 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:38.527596 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:38.528988 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.530083 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.532505 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.533131 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.533257 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:38.534912 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:38.534920 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:38.534927 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:38.534951 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:38.535002 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:38.535019 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:38.535100 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:38.535348 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 14:54:38.535489 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:54:38.537096 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:38.523977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45227 (local address 127.0.241.2:38163)
0619 14:54:38.525010 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:38.525053 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:38.527081 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:38.527846 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:38.527857 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:38.527976 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:38.528751 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:38.528780 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.530258 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.530266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:38.532367 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:38.532385 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.533356 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.533365 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:38.533720 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:38.534538 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:54:38.534544 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:38.534593 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:54:38.534704 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:38.535122 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:38.535135 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:38.535147 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 14:54:38.535329 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 14:54:38.535756 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":358,"threads_started":1}
I20260619 14:54:38.538419 21132 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45227
I20260619 14:54:38.538720 21132 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:38.539422 21132 heartbeater.cc:511] Master 127.0.241.62:45227 requested a full tablet report, sending...
I20260619 14:54:38.540762 20815 ts_manager.cc:194] Registered new tserver with Master: dfc926eab2ca4d5d8c90c39f629a447a (127.0.241.2:35323)
I20260619 14:54:38.541535 20815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.241.2:38163
W20260619 14:54:38.850435 21135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:38.850693 21135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:38.850855 21135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:38.858155 21135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:38.858297 21135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:38.858490 21135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.241.3
I20260619 14:54:38.867383 21135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.241.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.241.3
--webserver_port=0
--tserver_master_addrs=127.0.241.62:45227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.241.3
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:38.868656 21135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:38.870353 21135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:38.883011 21141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:38.883057 21143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:38.883083 21140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:38.883749 21135 server_base.cc:1034] running on GCE node
I20260619 14:54:38.884398 21135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:38.885303 21135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:38.886495 21135 hybrid_clock.cc:648] HybridClock initialized: now 1781880878886459 us; error 34 us; skew 500 ppm
I20260619 14:54:38.886799 21135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 14:54:38.889010 21135 webserver.cc:460] Webserver started at http://127.0.241.3:33931/ using document root <none> and password file <none>
I20260619 14:54:38.889694 21135 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:38.889810 21135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:38.890134 21135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:38.893090 21135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/data/instance:
uuid: "45cf8d76a01a496681b4bcc766893283"
format_stamp: "Formatted at 2026-06-19 14:54:38 on dist-test-slave-p3ml"
server_key: "2ad8eabd44a31ef66e51db1881e817ea"
server_key_iv: "c0fd2477e67437f285994eaebb08c192"
server_key_version: "encryptionkey@0"
I20260619 14:54:38.893911 21135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/wal/instance:
uuid: "45cf8d76a01a496681b4bcc766893283"
format_stamp: "Formatted at 2026-06-19 14:54:38 on dist-test-slave-p3ml"
server_key: "2ad8eabd44a31ef66e51db1881e817ea"
server_key_iv: "c0fd2477e67437f285994eaebb08c192"
server_key_version: "encryptionkey@0"
I20260619 14:54:38.899401 21135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 14:54:38.902719 21149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:38.904037 21135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:54:38.904188 21135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/wal
uuid: "45cf8d76a01a496681b4bcc766893283"
format_stamp: "Formatted at 2026-06-19 14:54:38 on dist-test-slave-p3ml"
server_key: "2ad8eabd44a31ef66e51db1881e817ea"
server_key_iv: "c0fd2477e67437f285994eaebb08c192"
server_key_version: "encryptionkey@0"
I20260619 14:54:38.904404 21135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:38.927187 21135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:38.928054 21135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:38.928329 21135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:38.929276 21135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:38.930685 21135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:38.930794 21135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:38.930914 21135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:38.931000 21135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:38.963122 21135 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.3:34487
I20260619 14:54:38.963212 21261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.3:34487 every 8 connection(s)
I20260619 14:54:38.964566 21135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/data/info.pb
I20260619 14:54:38.973881 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 21135
I20260619 14:54:38.974031 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/wal/instance
I20260619 14:54:38.974460 964 external_mini_cluster.cc:1428] Setting key 00f2c0976e8934dc447bf132abc23dc0
I20260619 14:54:38.976629 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:38.967074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:57379 (local address 127.0.241.62:45227)
0619 14:54:38.967257 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:38.967261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:38.968610 (+ 1349us) server_negotiation.cc:407] Connection header received
0619 14:54:38.970243 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:38.970247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:38.970312 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:38.970401 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:38.971361 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.972123 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.973606 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.974061 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.974207 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:38.975774 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:38.975780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:38.975785 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:38.975809 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:38.975839 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:38.975850 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:38.975956 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:38.976236 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 14:54:38.976325 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:54:38.977566 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:38.967636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45227 (local address 127.0.241.3:57379)
0619 14:54:38.968467 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:38.968500 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:38.970023 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:38.970602 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:38.970609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:38.970689 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:38.971225 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:38.971240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.972240 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.972244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:38.973400 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:38.973410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.974282 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.974288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:38.974528 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:38.975395 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:54:38.975417 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:38.975498 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:54:38.975609 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:38.975954 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:38.975965 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:38.975974 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:54:38.976177 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 14:54:38.976465 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":259,"threads_started":1}
I20260619 14:54:38.978688 21262 heartbeater.cc:347] Connected to a master server at 127.0.241.62:45227
I20260619 14:54:38.978965 21262 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:38.979772 21262 heartbeater.cc:511] Master 127.0.241.62:45227 requested a full tablet report, sending...
I20260619 14:54:38.981103 20815 ts_manager.cc:194] Registered new tserver with Master: 45cf8d76a01a496681b4bcc766893283 (127.0.241.3:34487)
I20260619 14:54:38.981833 20815 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.241.3:57379
I20260619 14:54:38.989181 964 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:54:38.998369 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:38.993112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.0.241.62:45227)
0619 14:54:38.993282 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:38.993286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:38.993734 (+ 448us) server_negotiation.cc:407] Connection header received
0619 14:54:38.993862 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:38.993865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:38.993939 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:38.994072 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:38.994517 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.995196 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.996499 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:38.996857 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:38.996955 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:38.997837 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:38.997842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:38.997845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:38.997861 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:38.997898 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:38.997910 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:38.997978 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:38.998084 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 14:54:38.998152 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:54:39.001770 20815 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:54:39.008317 20815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:54:39.033844 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.019281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.0.241.3:34487)
0619 14:54:39.020068 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:39.020078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:39.023354 (+ 3276us) server_negotiation.cc:407] Connection header received
0619 14:54:39.023700 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:39.023710 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:39.024014 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:39.024192 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:39.025407 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.027455 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.030792 (+ 3337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.032143 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.032319 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.032610 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:39.033268 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 14:54:39.033462 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":701,"threads_started":1}
I20260619 14:54:39.033946 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.019873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:44171 (local address 127.0.0.1:44836)
0619 14:54:39.023582 (+ 3709us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.023602 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:39.023735 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:39.024805 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:39.024813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:39.024853 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:39.025425 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:39.025440 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.027956 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.027965 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:39.030928 (+ 2963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:39.030940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.033051 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.033056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:39.033263 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.033294 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:54:39.033390 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:54:39.033576 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3526,"mutex_wait_us":346,"thread_start_us":2818,"threads_started":1}
I20260619 14:54:39.033995 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.020187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.0.241.1:44171)
0619 14:54:39.020917 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:39.020928 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:39.023886 (+ 2958us) server_negotiation.cc:407] Connection header received
0619 14:54:39.024002 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:39.024009 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:39.024256 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:39.024487 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:39.025671 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.027748 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.031135 (+ 3387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.032558 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.032700 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.032925 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:39.033462 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 14:54:39.033644 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":356,"threads_started":1}
I20260619 14:54:39.034789 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.019002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.3:34487 (local address 127.0.0.1:35330)
0619 14:54:39.023170 (+ 4168us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.023219 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:39.023459 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:39.024621 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:39.024632 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:39.024670 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:39.025149 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:39.025166 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.028107 (+ 2941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.028115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:39.030627 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:39.030638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.032748 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.032754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:39.032989 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.033054 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 14:54:39.034366 (+ 1312us) client_negotiation.cc:240] Negotiation successful
0619 14:54:39.034496 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3861,"mutex_wait_us":3,"thread_start_us":4230,"threads_started":1}
I20260619 14:54:39.037431 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.023648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:35323 (local address 127.0.0.1:34772)
0619 14:54:39.029626 (+ 5978us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.029643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:39.029848 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:39.030703 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:39.030709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:39.030734 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:39.031155 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:39.031165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.032564 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.032570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:39.034752 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:39.034768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.036794 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.036799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:39.036922 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.036944 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:54:39.037042 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:54:39.037159 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5849,"thread_start_us":5681,"threads_started":1}
I20260619 14:54:39.037647 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.024017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.0.241.2:35323)
0619 14:54:39.029170 (+ 5153us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:39.029180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:39.029740 (+ 560us) server_negotiation.cc:407] Connection header received
0619 14:54:39.030028 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:39.030036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:39.030246 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:39.030409 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:39.031285 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.032372 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.035020 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.036280 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.036408 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.036673 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:39.037093 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 14:54:39.037254 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5005,"thread_start_us":4897,"threads_started":1}
I20260619 14:54:39.037921 20936 tablet_service.cc:1467] Processing CreateTablet for tablet fef672f4416b463eab21d7fe8f68bd3c (DEFAULT_TABLE table=test-table [id=26134deec00041e898227893da0a1703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:39.038800 21197 tablet_service.cc:1467] Processing CreateTablet for tablet fef672f4416b463eab21d7fe8f68bd3c (DEFAULT_TABLE table=test-table [id=26134deec00041e898227893da0a1703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:39.039376 20936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef672f4416b463eab21d7fe8f68bd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:39.040134 21197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef672f4416b463eab21d7fe8f68bd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:39.040740 21067 tablet_service.cc:1467] Processing CreateTablet for tablet fef672f4416b463eab21d7fe8f68bd3c (DEFAULT_TABLE table=test-table [id=26134deec00041e898227893da0a1703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:39.041698 21067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef672f4416b463eab21d7fe8f68bd3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:39.059863 21280 tablet_bootstrap.cc:492] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a: Bootstrap starting.
I20260619 14:54:39.060312 21281 tablet_bootstrap.cc:492] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283: Bootstrap starting.
I20260619 14:54:39.060514 21282 tablet_bootstrap.cc:492] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76: Bootstrap starting.
I20260619 14:54:39.064721 21280 tablet_bootstrap.cc:654] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:39.064872 21281 tablet_bootstrap.cc:654] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:39.065699 21282 tablet_bootstrap.cc:654] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:39.065801 21280 log.cc:824] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:39.066058 21281 log.cc:824] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:39.066903 21282 log.cc:824] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:39.068771 21280 tablet_bootstrap.cc:492] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a: No bootstrap required, opened a new log
I20260619 14:54:39.069039 21280 ts_tablet_manager.cc:1397] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 14:54:39.070334 21281 tablet_bootstrap.cc:492] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283: No bootstrap required, opened a new log
I20260619 14:54:39.070629 21281 ts_tablet_manager.cc:1397] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 14:54:39.074276 21282 tablet_bootstrap.cc:492] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76: No bootstrap required, opened a new log
I20260619 14:54:39.074525 21282 ts_tablet_manager.cc:1397] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260619 14:54:39.075201 21001 heartbeater.cc:503] Master 127.0.241.62:45227 was elected leader, sending a full tablet report...
I20260619 14:54:39.077560 21280 raft_consensus.cc:348] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf8d76a01a496681b4bcc766893283" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 34487 } } peers { permanent_uuid: "2a4e7c79039c4b0eaed120a55a7abd76" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 44171 } } peers { permanent_uuid: "dfc926eab2ca4d5d8c90c39f629a447a" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 35323 } }
I20260619 14:54:39.078118 21280 raft_consensus.cc:374] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:39.078276 21280 raft_consensus.cc:729] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc926eab2ca4d5d8c90c39f629a447a, State: Initialized, Role: FOLLOWER
I20260619 14:54:39.079088 21280 consensus_queue.cc:260] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf8d76a01a496681b4bcc766893283" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 34487 } } peers { permanent_uuid: "2a4e7c79039c4b0eaed120a55a7abd76" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 44171 } } peers { permanent_uuid: "dfc926eab2ca4d5d8c90c39f629a447a" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 35323 } }
I20260619 14:54:39.081015 21281 raft_consensus.cc:348] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf8d76a01a496681b4bcc766893283" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 34487 } } peers { permanent_uuid: "2a4e7c79039c4b0eaed120a55a7abd76" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 44171 } } peers { permanent_uuid: "dfc926eab2ca4d5d8c90c39f629a447a" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 35323 } }
I20260619 14:54:39.081463 21281 raft_consensus.cc:374] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:39.081588 21281 raft_consensus.cc:729] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cf8d76a01a496681b4bcc766893283, State: Initialized, Role: FOLLOWER
I20260619 14:54:39.081792 21132 heartbeater.cc:503] Master 127.0.241.62:45227 was elected leader, sending a full tablet report...
I20260619 14:54:39.082464 21280 ts_tablet_manager.cc:1428] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260619 14:54:39.082317 21281 consensus_queue.cc:260] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf8d76a01a496681b4bcc766893283" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 34487 } } peers { permanent_uuid: "2a4e7c79039c4b0eaed120a55a7abd76" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 44171 } } peers { permanent_uuid: "dfc926eab2ca4d5d8c90c39f629a447a" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 35323 } }
I20260619 14:54:39.083364 21262 heartbeater.cc:503] Master 127.0.241.62:45227 was elected leader, sending a full tablet report...
I20260619 14:54:39.084614 21287 raft_consensus.cc:482] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:39.084898 21287 raft_consensus.cc:504] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf8d76a01a496681b4bcc766893283" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 34487 } } peers { permanent_uuid: "2a4e7c79039c4b0eaed120a55a7abd76" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 44171 } } peers { permanent_uuid: "dfc926eab2ca4d5d8c90c39f629a447a" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 35323 } }
I20260619 14:54:39.086901 21282 raft_consensus.cc:348] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf8d76a01a496681b4bcc766893283" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 34487 } } peers { permanent_uuid: "2a4e7c79039c4b0eaed120a55a7abd76" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 44171 } } peers { permanent_uuid: "dfc926eab2ca4d5d8c90c39f629a447a" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 35323 } }
I20260619 14:54:39.087525 21282 raft_consensus.cc:374] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:39.087693 21282 raft_consensus.cc:729] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4e7c79039c4b0eaed120a55a7abd76, State: Initialized, Role: FOLLOWER
I20260619 14:54:39.087777 21287 leader_election.cc:288] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a4e7c79039c4b0eaed120a55a7abd76 (127.0.241.1:44171), dfc926eab2ca4d5d8c90c39f629a447a (127.0.241.2:35323)
I20260619 14:54:39.088608 21282 consensus_queue.cc:260] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf8d76a01a496681b4bcc766893283" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 34487 } } peers { permanent_uuid: "2a4e7c79039c4b0eaed120a55a7abd76" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 44171 } } peers { permanent_uuid: "dfc926eab2ca4d5d8c90c39f629a447a" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 35323 } }
I20260619 14:54:39.090219 21281 ts_tablet_manager.cc:1428] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283: Time spent starting tablet: real 0.019s user 0.000s sys 0.012s
I20260619 14:54:39.090389 21282 ts_tablet_manager.cc:1428] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20260619 14:54:39.096115 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.087639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:37733 (local address 127.0.241.2:35323)
0619 14:54:39.088367 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:39.088386 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:39.088407 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:54:39.088467 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:39.088471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:39.088539 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:39.088626 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:39.089075 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.090140 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.093883 (+ 3743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.095046 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.095176 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.095272 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:39.095724 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 14:54:39.095846 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598}
I20260619 14:54:39.096144 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.087151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.2:35323 (local address 127.0.241.3:37733)
0619 14:54:39.087838 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.087863 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:39.087985 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:39.088673 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:39.088680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:39.088706 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:39.088908 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:39.088921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.090335 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.090340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:39.092408 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:39.092427 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.095200 (+ 2773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.095206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:39.095466 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.095502 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:54:39.095609 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:54:39.095819 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":36}
I20260619 14:54:39.097160 21087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef672f4416b463eab21d7fe8f68bd3c" candidate_uuid: "45cf8d76a01a496681b4bcc766893283" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc926eab2ca4d5d8c90c39f629a447a" is_pre_election: true
I20260619 14:54:39.097617 21087 raft_consensus.cc:2454] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45cf8d76a01a496681b4bcc766893283 in term 0.
I20260619 14:54:39.098242 21151 leader_election.cc:302] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45cf8d76a01a496681b4bcc766893283, dfc926eab2ca4d5d8c90c39f629a447a; no voters:
I20260619 14:54:39.098628 21288 raft_consensus.cc:2781] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:54:39.098742 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.091517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.241.3:33893 (local address 127.0.241.1:44171)
0619 14:54:39.091833 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:39.091840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:39.093201 (+ 1361us) server_negotiation.cc:407] Connection header received
0619 14:54:39.093519 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:39.093529 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:39.093594 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:39.093690 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:39.094245 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.095069 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.097044 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.097904 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.098032 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.098118 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:39.098384 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 14:54:39.098500 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260619 14:54:39.098778 21288 raft_consensus.cc:482] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:54:39.098906 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.091497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:44171 (local address 127.0.241.3:33893)
0619 14:54:39.093056 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.093097 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:39.093325 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:39.093800 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:39.093805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:39.093832 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:39.094105 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:39.094112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.095214 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.095219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:39.096881 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:39.096892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.098058 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.098063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:39.098217 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.098243 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:54:39.098361 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:54:39.098479 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"mutex_wait_us":2,"thread_start_us":1305,"threads_started":1}
I20260619 14:54:39.098949 21288 raft_consensus.cc:3037] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:39.099785 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef672f4416b463eab21d7fe8f68bd3c" candidate_uuid: "45cf8d76a01a496681b4bcc766893283" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a4e7c79039c4b0eaed120a55a7abd76" is_pre_election: true
I20260619 14:54:39.100219 20956 raft_consensus.cc:2454] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45cf8d76a01a496681b4bcc766893283 in term 0.
I20260619 14:54:39.101320 21288 raft_consensus.cc:504] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf8d76a01a496681b4bcc766893283" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 34487 } } peers { permanent_uuid: "2a4e7c79039c4b0eaed120a55a7abd76" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 44171 } } peers { permanent_uuid: "dfc926eab2ca4d5d8c90c39f629a447a" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 35323 } }
I20260619 14:54:39.101938 21288 leader_election.cc:288] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [CANDIDATE]: Term 1 election: Requested vote from peers 2a4e7c79039c4b0eaed120a55a7abd76 (127.0.241.1:44171), dfc926eab2ca4d5d8c90c39f629a447a (127.0.241.2:35323)
I20260619 14:54:39.102326 21087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef672f4416b463eab21d7fe8f68bd3c" candidate_uuid: "45cf8d76a01a496681b4bcc766893283" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc926eab2ca4d5d8c90c39f629a447a"
I20260619 14:54:39.102356 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef672f4416b463eab21d7fe8f68bd3c" candidate_uuid: "45cf8d76a01a496681b4bcc766893283" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a4e7c79039c4b0eaed120a55a7abd76"
I20260619 14:54:39.102520 20956 raft_consensus.cc:3037] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:39.102526 21087 raft_consensus.cc:3037] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:39.104640 20956 raft_consensus.cc:2454] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45cf8d76a01a496681b4bcc766893283 in term 1.
I20260619 14:54:39.104746 21087 raft_consensus.cc:2454] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45cf8d76a01a496681b4bcc766893283 in term 1.
I20260619 14:54:39.105017 21153 leader_election.cc:302] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a4e7c79039c4b0eaed120a55a7abd76, 45cf8d76a01a496681b4bcc766893283; no voters:
I20260619 14:54:39.105317 21288 raft_consensus.cc:2781] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:39.105571 21288 raft_consensus.cc:686] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 1 LEADER]: Becoming Leader. State: Replica: 45cf8d76a01a496681b4bcc766893283, State: Running, Role: LEADER
I20260619 14:54:39.106074 21288 consensus_queue.cc:237] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf8d76a01a496681b4bcc766893283" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 34487 } } peers { permanent_uuid: "2a4e7c79039c4b0eaed120a55a7abd76" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 44171 } } peers { permanent_uuid: "dfc926eab2ca4d5d8c90c39f629a447a" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 35323 } }
I20260619 14:54:39.110759 20814 catalog_manager.cc:5495] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45cf8d76a01a496681b4bcc766893283 (127.0.241.3). New cstate: current_term: 1 leader_uuid: "45cf8d76a01a496681b4bcc766893283" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf8d76a01a496681b4bcc766893283" member_type: VOTER last_known_addr { host: "127.0.241.3" port: 34487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a4e7c79039c4b0eaed120a55a7abd76" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 44171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfc926eab2ca4d5d8c90c39f629a447a" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 35323 } health_report { overall_health: UNKNOWN } } }
I20260619 14:54:39.132296 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.127635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35334 (local address 127.0.241.3:34487)
0619 14:54:39.127799 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:39.127803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:39.127820 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:54:39.127976 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:39.127980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:39.128052 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:39.128174 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:39.128474 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.129188 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.130464 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.130850 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.130945 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.131337 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:39.131868 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:54:39.131986 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:39.132059 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 14:54:39.132116 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:54:39.139739 20956 raft_consensus.cc:1261] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76 [term 1 FOLLOWER]: Refusing update from remote peer 45cf8d76a01a496681b4bcc766893283: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:39.139739 21087 raft_consensus.cc:1261] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a [term 1 FOLLOWER]: Refusing update from remote peer 45cf8d76a01a496681b4bcc766893283: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:39.140628 21288 consensus_queue.cc:1035] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a4e7c79039c4b0eaed120a55a7abd76" member_type: VOTER last_known_addr { host: "127.0.241.1" port: 44171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:39.140888 21287 consensus_queue.cc:1035] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfc926eab2ca4d5d8c90c39f629a447a" member_type: VOTER last_known_addr { host: "127.0.241.2" port: 35323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:39.151613 21293 mvcc.cc:205] Tried to move back new op lower bound from 7298584080947326976 to 7298584080822812672. Current Snapshot: MvccSnapshot[applied={T|T < 7298584080947326976}]
I20260619 14:54:39.154610 21295 mvcc.cc:205] Tried to move back new op lower bound from 7298584080947326976 to 7298584080822812672. Current Snapshot: MvccSnapshot[applied={T|T < 7298584080947326976}]
I20260619 14:54:39.176935 20813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:39.177335 20813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:39.180791 20813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a340ffb4b9434d528cc6737258a34c0f: Sending DeleteTablet for 3 replicas of tablet fef672f4416b463eab21d7fe8f68bd3c
I20260619 14:54:39.181859 20936 tablet_service.cc:1514] Processing DeleteTablet for tablet fef672f4416b463eab21d7fe8f68bd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:39 UTC) from {username='slave'} at 127.0.0.1:44836
I20260619 14:54:39.181876 21197 tablet_service.cc:1514] Processing DeleteTablet for tablet fef672f4416b463eab21d7fe8f68bd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:39 UTC) from {username='slave'} at 127.0.0.1:35330
I20260619 14:54:39.182083 21067 tablet_service.cc:1514] Processing DeleteTablet for tablet fef672f4416b463eab21d7fe8f68bd3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:39 UTC) from {username='slave'} at 127.0.0.1:34772
I20260619 14:54:39.184680 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 20874
I20260619 14:54:39.194330 21304 tablet_replica.cc:330] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76: stopping tablet replica
I20260619 14:54:39.195096 21304 raft_consensus.cc:2229] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:39.195850 21304 raft_consensus.cc:2258] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:39.196805 21305 tablet_replica.cc:330] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a: stopping tablet replica
I20260619 14:54:39.197463 21305 raft_consensus.cc:2229] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:39.199172 21304 ts_tablet_manager.cc:1905] T fef672f4416b463eab21d7fe8f68bd3c P 2a4e7c79039c4b0eaed120a55a7abd76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:39.205996 21305 raft_consensus.cc:2258] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:39.208581 21305 ts_tablet_manager.cc:1905] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:39.209229 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.186437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.0.241.1:44171)
0619 14:54:39.186654 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:39.186664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:39.200022 (+ 13358us) server_negotiation.cc:407] Connection header received
0619 14:54:39.200315 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:39.200321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:39.200438 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:39.200519 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:39.201260 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.202479 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.204512 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.205228 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.205336 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.207457 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:39.207479 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:39.207497 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:39.208253 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:39.208314 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:39.208339 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:39.208430 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:39.208746 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 14:54:39.208860 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:54:39.210381 20996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:39.219249 21303 tablet_replica.cc:330] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283: stopping tablet replica
I20260619 14:54:39.219676 21305 ts_tablet_manager.cc:1918] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:39.219700 21303 raft_consensus.cc:2229] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:54:39.219853 21305 log.cc:1196] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-1/wal/wals/fef672f4416b463eab21d7fe8f68bd3c
I20260619 14:54:39.220378 21303 raft_consensus.cc:2258] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:39.220512 21305 ts_tablet_manager.cc:1939] T fef672f4416b463eab21d7fe8f68bd3c P dfc926eab2ca4d5d8c90c39f629a447a: Deleting consensus metadata
I20260619 14:54:39.223534 21303 ts_tablet_manager.cc:1905] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:39.224339 20800 catalog_manager.cc:4841] TS dfc926eab2ca4d5d8c90c39f629a447a (127.0.241.2:35323): tablet fef672f4416b463eab21d7fe8f68bd3c (table test-table [id=26134deec00041e898227893da0a1703]) successfully deleted
I20260619 14:54:39.231276 21303 ts_tablet_manager.cc:1918] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:39.231582 21303 log.cc:1196] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283: Deleting WAL directory at /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880565277815-964-0/minicluster-data/ts-2/wal/wals/fef672f4416b463eab21d7fe8f68bd3c
I20260619 14:54:39.232319 21303 ts_tablet_manager.cc:1939] T fef672f4416b463eab21d7fe8f68bd3c P 45cf8d76a01a496681b4bcc766893283: Deleting consensus metadata
I20260619 14:54:39.234565 20800 catalog_manager.cc:4841] TS 45cf8d76a01a496681b4bcc766893283 (127.0.241.3:34487): tablet fef672f4416b463eab21d7fe8f68bd3c (table test-table [id=26134deec00041e898227893da0a1703]) successfully deleted
I20260619 14:54:39.319023 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 20874
I20260619 14:54:39.337250 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21005
W20260619 14:54:39.337456 20802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.241.1:44171 (error 108)
I20260619 14:54:39.339614 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.338972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:44171 (local address 127.0.0.1:44846)
0619 14:54:39.339261 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.339402 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260619 14:54:39.339895 20802 catalog_manager.cc:4584] TS 2a4e7c79039c4b0eaed120a55a7abd76 (127.0.241.1:44171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fef672f4416b463eab21d7fe8f68bd3c: Network error: Client connection negotiation failed: client connection to 127.0.241.1:44171: connect: Connection refused (error 111)
I20260619 14:54:39.345332 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.338312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.0.241.2:35323)
0619 14:54:39.338554 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:39.338562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:39.338663 (+ 101us) server_negotiation.cc:407] Connection header received
0619 14:54:39.338872 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:39.338876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:39.338957 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:39.339061 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:39.339532 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.340493 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.341786 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.342174 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.342313 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.343715 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:39.343724 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:39.343738 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:39.344335 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:39.344389 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:39.344409 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:39.344534 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:39.344745 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 14:54:39.344870 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 14:54:39.346064 21127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:39.393805 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.393229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:44171 (local address 127.0.0.1:44854)
0619 14:54:39.393472 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.393585 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 14:54:39.394929 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.394522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:44171 (local address 127.0.0.1:44864)
0619 14:54:39.394686 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.394777 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 14:54:39.436506 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 21005
I20260619 14:54:39.453461 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21135
I20260619 14:54:39.455477 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.454987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:44171 (local address 127.0.0.1:44880)
0619 14:54:39.455186 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.455258 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 14:54:39.456655 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.456265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:44171 (local address 127.0.0.1:44888)
0619 14:54:39.456437 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.456523 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 14:54:39.460994 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.454387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.0.241.3:34487)
0619 14:54:39.454571 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:39.454577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:39.454601 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:54:39.454677 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:39.454681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:39.454774 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:39.454883 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:39.455391 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.456545 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.457862 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:39.458375 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:39.458509 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:39.459653 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:39.459661 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:39.459676 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:39.460283 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:39.460328 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:39.460347 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:39.460465 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:39.460580 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 14:54:39.460670 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:54:39.461727 21257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:39.538252 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.537722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:44171 (local address 127.0.0.1:44902)
0619 14:54:39.538000 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.538106 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260619 14:54:39.539299 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.538931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:44171 (local address 127.0.0.1:44912)
0619 14:54:39.539121 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.539197 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 14:54:39.584149 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 21135
I20260619 14:54:39.603626 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 20784
I20260619 14:54:39.604619 20845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:39.712841 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:39.712343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.1:44171 (local address 127.0.0.1:44922)
0619 14:54:39.712527 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:39.712616 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 14:54:39.712822 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 20784
2026-06-19T14:54:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2721 ms)
[----------] 6 tests from EncryptionPolicyTest (21277 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:39 dist-test-slave-p3ml krb5kdc[21314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:39 dist-test-slave-p3ml krb5kdc[21314](info): set up 2 sockets
Jun 19 14:54:39 dist-test-slave-p3ml krb5kdc[21314](info): commencing operation
krb5kdc: starting...
W20260619 14:54:40.815805 964 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:40 dist-test-slave-p3ml krb5kdc[21314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:54:41.054725 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:41.385146 21327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:41.385404 21327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:41.385524 21327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:41.392750 21327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:41.392876 21327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:41.392975 21327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:41.393085 21327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:41.393158 21327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:41.402981 21327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:41.404489 21327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:41.405830 21327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:41.417912 21332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:41.417909 21333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:41.418164 21335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:41.418805 21327 server_base.cc:1034] running on GCE node
I20260619 14:54:41.419662 21327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:41.421054 21327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:41.422327 21327 hybrid_clock.cc:648] HybridClock initialized: now 1781880881422268 us; error 42 us; skew 500 ppm
I20260619 14:54:41.422688 21327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:41 dist-test-slave-p3ml krb5kdc[21314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:41.428985 21327 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:54:41.430392 21327 webserver.cc:460] Webserver started at http://127.0.0.1:36243/ using document root <none> and password file <none>
I20260619 14:54:41.431018 21327 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:41.431114 21327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:41.431365 21327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:41.434228 21327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "83dcd3311d2e4342aa3c7eafe793ab99"
format_stamp: "Formatted at 2026-06-19 14:54:41 on dist-test-slave-p3ml"
server_key: "f29da39ac2e14eac381b7021f2dfc6d1"
server_key_iv: "12875d263bc39afadf05d819828ed3af"
server_key_version: "encryptionkey@0"
I20260619 14:54:41.434931 21327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "83dcd3311d2e4342aa3c7eafe793ab99"
format_stamp: "Formatted at 2026-06-19 14:54:41 on dist-test-slave-p3ml"
server_key: "f29da39ac2e14eac381b7021f2dfc6d1"
server_key_iv: "12875d263bc39afadf05d819828ed3af"
server_key_version: "encryptionkey@0"
I20260619 14:54:41.439824 21327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 14:54:41.442905 21342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:41.444410 21327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:54:41.444591 21327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "83dcd3311d2e4342aa3c7eafe793ab99"
format_stamp: "Formatted at 2026-06-19 14:54:41 on dist-test-slave-p3ml"
server_key: "f29da39ac2e14eac381b7021f2dfc6d1"
server_key_iv: "12875d263bc39afadf05d819828ed3af"
server_key_version: "encryptionkey@0"
I20260619 14:54:41.444788 21327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:41.478811 21327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:41.483394 21327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:41.483716 21327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:41.502046 21327 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33799
I20260619 14:54:41.502070 21393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33799 every 8 connection(s)
I20260619 14:54:41.503458 21327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:41.506958 21394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:41.507702 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 21327
I20260619 14:54:41.507927 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:41.508774 964 external_mini_cluster.cc:1428] Setting key d8b789b0e8cb648612315a0bd8f5ecfb
I20260619 14:54:41.520764 21394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99: Bootstrap starting.
Jun 19 14:54:41 dist-test-slave-p3ml krb5kdc[21314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:54:41.524394 21394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:41.525422 21394 log.cc:824] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:41.528252 21394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99: No bootstrap required, opened a new log
I20260619 14:54:41.532043 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:41.510703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55821 (local address 127.0.0.1:33799)
0619 14:54:41.511595 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:41.511607 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:41.511647 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:54:41.513327 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:41.513347 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:41.513723 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:41.514046 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:41.515778 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:41.517041 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:41.518883 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:41.519576 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:41.519703 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:41.524453 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:41.524460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:41.524473 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:41.524537 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:41.527738 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:41.528561 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:41.528567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:41.528572 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:41.528750 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:41.529224 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:41.529229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:41.529231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:41.529760 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:41.530097 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:41.530715 (+ 618us) server_negotiation.cc:299] Negotiation successful
0619 14:54:41.531027 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":366,"threads_started":1}
I20260619 14:54:41.535872 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:54:41.536356 21394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dcd3311d2e4342aa3c7eafe793ab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33799 } }
I20260619 14:54:41.536700 21394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:41.536798 21394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83dcd3311d2e4342aa3c7eafe793ab99, State: Initialized, Role: FOLLOWER
I20260619 14:54:41.537309 21394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dcd3311d2e4342aa3c7eafe793ab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33799 } }
I20260619 14:54:41.537545 21394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:41.537683 21394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:41.537858 21394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:41.540251 21394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dcd3311d2e4342aa3c7eafe793ab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33799 } }
I20260619 14:54:41.540791 21394 leader_election.cc:302] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83dcd3311d2e4342aa3c7eafe793ab99; no voters:
I20260619 14:54:41.541458 21394 leader_election.cc:288] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:41.541555 21404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:41.543959 21394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:41.550640 21404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [term 1 LEADER]: Becoming Leader. State: Replica: 83dcd3311d2e4342aa3c7eafe793ab99, State: Running, Role: LEADER
I20260619 14:54:41.551419 21404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dcd3311d2e4342aa3c7eafe793ab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33799 } }
I20260619 14:54:41.556509 21406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83dcd3311d2e4342aa3c7eafe793ab99. Latest consensus state: current_term: 1 leader_uuid: "83dcd3311d2e4342aa3c7eafe793ab99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dcd3311d2e4342aa3c7eafe793ab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33799 } } }
I20260619 14:54:41.556636 21405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83dcd3311d2e4342aa3c7eafe793ab99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83dcd3311d2e4342aa3c7eafe793ab99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33799 } } }
I20260619 14:54:41.556945 21406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:41.557026 21405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:41.569327 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:41.540824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36027 (local address 127.0.0.1:33799)
0619 14:54:41.541006 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:41.541012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:41.547605 (+ 6593us) server_negotiation.cc:407] Connection header received
0619 14:54:41.547902 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:41.547910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:41.548031 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:41.548152 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:41.550563 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:41.552057 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:41.554140 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:41.555108 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:41.555287 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:41.558478 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:41.558488 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:41.558496 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:41.558541 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:41.563579 (+ 5038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:41.564920 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:41.564929 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:41.564936 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:41.565125 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:41.565895 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:41.565903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:41.565908 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:41.566440 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:41.568300 (+ 1860us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:41.568530 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 14:54:41.568775 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 14:54:41.571569 21418 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:54:41.571962 21418 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:54:41.576182 21419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:41.578490 21419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:41.586870 21419 catalog_manager.cc:1347] Generated new cluster ID: 688afcdfc9c04254b5bf242bd6e334f8
I20260619 14:54:41.587037 21419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:41.619601 21419 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:41.621115 21419 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:41.631484 21419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83dcd3311d2e4342aa3c7eafe793ab99: Generated new TSK 0
I20260619 14:54:41.632419 21419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:41.648753 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21327
I20260619 14:54:41.650223 21389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:41.744299 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 21327
W20260619 14:54:41.762410 21312 connection.cc:535] client connection to 127.0.0.1:33799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33799): Connection reset by peer (error 104)
2026-06-19T14:54:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2028 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:41 dist-test-slave-p3ml krb5kdc[21426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:41 dist-test-slave-p3ml krb5kdc[21426](info): set up 2 sockets
Jun 19 14:54:41 dist-test-slave-p3ml krb5kdc[21426](info): commencing operation
krb5kdc: starting...
W20260619 14:54:42.839766 964 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:42 dist-test-slave-p3ml krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:54:43.086215 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:43.413074 21439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:43.413475 21439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:43.413635 21439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:43.420954 21439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:43.421049 21439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:43.421094 21439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:43.421172 21439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:43.421250 21439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:43.430789 21439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40043
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:43.432016 21439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:43.433236 21439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:43.444697 21447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:43.444823 21445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:43.445322 21444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:43.445607 21439 server_base.cc:1034] running on GCE node
I20260619 14:54:43.446319 21439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:43.447661 21439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:43.448881 21439 hybrid_clock.cc:648] HybridClock initialized: now 1781880883448840 us; error 39 us; skew 500 ppm
I20260619 14:54:43.449213 21439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:43 dist-test-slave-p3ml krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:43.452418 21439 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:54:43.453698 21439 webserver.cc:460] Webserver started at http://127.0.0.1:43551/ using document root <none> and password file <none>
I20260619 14:54:43.454278 21439 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:43.454375 21439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:43.454628 21439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:43.457424 21439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "f94e6596b88e4dcca2d08a8bae1beb4c"
format_stamp: "Formatted at 2026-06-19 14:54:43 on dist-test-slave-p3ml"
server_key: "fb40cf621ed64d43de9cd94f630c216b"
server_key_iv: "3f7563e9accb6534bd290aa2d4f9e13d"
server_key_version: "encryptionkey@0"
I20260619 14:54:43.458117 21439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "f94e6596b88e4dcca2d08a8bae1beb4c"
format_stamp: "Formatted at 2026-06-19 14:54:43 on dist-test-slave-p3ml"
server_key: "fb40cf621ed64d43de9cd94f630c216b"
server_key_iv: "3f7563e9accb6534bd290aa2d4f9e13d"
server_key_version: "encryptionkey@0"
I20260619 14:54:43.462886 21439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:54:43.466099 21454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:43.467286 21439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:43.467506 21439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "f94e6596b88e4dcca2d08a8bae1beb4c"
format_stamp: "Formatted at 2026-06-19 14:54:43 on dist-test-slave-p3ml"
server_key: "fb40cf621ed64d43de9cd94f630c216b"
server_key_iv: "3f7563e9accb6534bd290aa2d4f9e13d"
server_key_version: "encryptionkey@0"
I20260619 14:54:43.467694 21439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:43.484129 21439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:43.488081 21439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:43.488353 21439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:43.507550 21439 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40043
I20260619 14:54:43.507629 21505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40043 every 8 connection(s)
I20260619 14:54:43.508891 21439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:43.510597 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 21439
I20260619 14:54:43.510789 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:43.511235 964 external_mini_cluster.cc:1428] Setting key d16ae54834fc6769f4b6f36549260b41
I20260619 14:54:43.513000 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:43.525758 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:43.512995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44331 (local address 127.0.0.1:40043)
0619 14:54:43.513768 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:43.513782 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:43.513828 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:54:43.515959 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:43.515994 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:43.516492 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:43.516916 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:43.517435 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:43.519477 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:43.520735 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:43.521772 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:43.521937 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:43.523081 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:43.523091 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:43.523107 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:43.523866 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:43.523956 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:43.524000 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:43.524167 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:43.524362 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 14:54:43.524529 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":303,"threads_started":1}
I20260619 14:54:43.527180 21506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c: Bootstrap starting.
I20260619 14:54:43.529318 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:54:43.530586 21506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:43.531769 21506 log.cc:824] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:43.534262 21506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c: No bootstrap required, opened a new log
I20260619 14:54:43.541214 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:43.534615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58337 (local address 127.0.0.1:40043)
0619 14:54:43.534837 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:43.534844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:43.535197 (+ 353us) server_negotiation.cc:407] Connection header received
0619 14:54:43.535371 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:43.535375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:43.535470 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:43.535562 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:43.536129 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:43.536922 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:43.538792 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:43.539199 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:43.539294 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:43.540439 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:43.540443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:43.540447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:43.540462 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:43.540512 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:43.540529 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:43.540620 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:43.540834 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 14:54:43.540872 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:54:43.543708 21506 raft_consensus.cc:348] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e6596b88e4dcca2d08a8bae1beb4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40043 } }
I20260619 14:54:43.544059 21506 raft_consensus.cc:374] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:43.544158 21506 raft_consensus.cc:729] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94e6596b88e4dcca2d08a8bae1beb4c, State: Initialized, Role: FOLLOWER
I20260619 14:54:43.544720 21506 consensus_queue.cc:260] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e6596b88e4dcca2d08a8bae1beb4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40043 } }
I20260619 14:54:43.544957 21506 raft_consensus.cc:388] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:43.545096 21506 raft_consensus.cc:482] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:43.545286 21506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:43.547351 21506 raft_consensus.cc:504] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e6596b88e4dcca2d08a8bae1beb4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40043 } }
I20260619 14:54:43.547844 21506 leader_election.cc:302] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f94e6596b88e4dcca2d08a8bae1beb4c; no voters:
I20260619 14:54:43.548396 21506 leader_election.cc:288] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:43.548529 21516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:43.549239 21516 raft_consensus.cc:686] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [term 1 LEADER]: Becoming Leader. State: Replica: f94e6596b88e4dcca2d08a8bae1beb4c, State: Running, Role: LEADER
I20260619 14:54:43.549852 21516 consensus_queue.cc:237] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e6596b88e4dcca2d08a8bae1beb4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40043 } }
I20260619 14:54:43.550383 21506 sys_catalog.cc:564] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:43.553455 21518 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f94e6596b88e4dcca2d08a8bae1beb4c. Latest consensus state: current_term: 1 leader_uuid: "f94e6596b88e4dcca2d08a8bae1beb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e6596b88e4dcca2d08a8bae1beb4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40043 } } }
I20260619 14:54:43.553267 21517 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f94e6596b88e4dcca2d08a8bae1beb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f94e6596b88e4dcca2d08a8bae1beb4c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40043 } } }
I20260619 14:54:43.553980 21518 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:43.554035 21517 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:43.556367 21524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:43.561836 21524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:43.573221 21524 catalog_manager.cc:1347] Generated new cluster ID: f6f8480490b24690bfe78a04e282cca2
I20260619 14:54:43.573411 21524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:43.596756 21524 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:43.598526 21524 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:43.605046 21524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f94e6596b88e4dcca2d08a8bae1beb4c: Generated new TSK 0
I20260619 14:54:43.605808 21524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:43.613250 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21439
W20260619 14:54:43.614562 21501 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44331
W20260619 14:54:43.615331 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21439: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:54:43.615556 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 21439
W20260619 14:54:43.632479 21424 connection.cc:535] client connection to 127.0.0.1:40043 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40043): Connection reset by peer (error 104)
2026-06-19T14:54:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:43 dist-test-slave-p3ml krb5kdc[21537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:43 dist-test-slave-p3ml krb5kdc[21537](info): set up 2 sockets
Jun 19 14:54:43 dist-test-slave-p3ml krb5kdc[21537](info): commencing operation
krb5kdc: starting...
W20260619 14:54:44.731611 964 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.009s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:44 dist-test-slave-p3ml krb5kdc[21537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:54:44.962131 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:45.291448 21550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:45.291769 21550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:45.291939 21550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:45.298949 21550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:45.299093 21550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:45.299196 21550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:45.299322 21550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:45.299397 21550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:45.308823 21550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:45.310042 21550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:45.311277 21550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:45.322918 21555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:45.322835 21558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:45.322886 21556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:45.323660 21550 server_base.cc:1034] running on GCE node
I20260619 14:54:45.324467 21550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:45.325723 21550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:45.326948 21550 hybrid_clock.cc:648] HybridClock initialized: now 1781880885326911 us; error 45 us; skew 500 ppm
I20260619 14:54:45.327323 21550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:45 dist-test-slave-p3ml krb5kdc[21537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:45.330232 21550 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:54:45.331744 21550 webserver.cc:460] Webserver started at http://127.0.0.1:46739/ using document root <none> and password file <none>
I20260619 14:54:45.332396 21550 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:45.332537 21550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:45.332836 21550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:45.335731 21550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "cc6d2ceb96a34d029236f18b4e5b43ec"
format_stamp: "Formatted at 2026-06-19 14:54:45 on dist-test-slave-p3ml"
server_key: "371bf172d86be059becfd51ff150e5e7"
server_key_iv: "bf452b124fd3e85c72e53f697fdc4fdf"
server_key_version: "encryptionkey@0"
I20260619 14:54:45.336459 21550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "cc6d2ceb96a34d029236f18b4e5b43ec"
format_stamp: "Formatted at 2026-06-19 14:54:45 on dist-test-slave-p3ml"
server_key: "371bf172d86be059becfd51ff150e5e7"
server_key_iv: "bf452b124fd3e85c72e53f697fdc4fdf"
server_key_version: "encryptionkey@0"
I20260619 14:54:45.341362 21550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:54:45.344671 21565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:45.345882 21550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:54:45.346065 21550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "cc6d2ceb96a34d029236f18b4e5b43ec"
format_stamp: "Formatted at 2026-06-19 14:54:45 on dist-test-slave-p3ml"
server_key: "371bf172d86be059becfd51ff150e5e7"
server_key_iv: "bf452b124fd3e85c72e53f697fdc4fdf"
server_key_version: "encryptionkey@0"
I20260619 14:54:45.346259 21550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:45.352066 21550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:45.356297 21550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:45.356587 21550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:45.373426 21550 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36311
I20260619 14:54:45.373463 21616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36311 every 8 connection(s)
I20260619 14:54:45.374783 21550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:45.378328 21617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:45.384177 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 21550
I20260619 14:54:45.384375 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:45.384850 964 external_mini_cluster.cc:1428] Setting key 1d31db58f241ca7394e5ff35db7acfcd
I20260619 14:54:45.389233 21617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec: Bootstrap starting.
I20260619 14:54:45.393018 21617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:45.393257 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:45.386507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46601 (local address 127.0.0.1:36311)
0619 14:54:45.387293 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:45.387306 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:45.387349 (+ 43us) server_negotiation.cc:407] Connection header received
0619 14:54:45.389695 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:45.389727 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:45.390124 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:45.390516 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:45.391521 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:45.391528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:45.391543 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:45.391560 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:45.391622 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:45.391654 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:45.391848 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:45.391964 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 14:54:45.392140 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":317,"threads_started":1}
I20260619 14:54:45.394089 21617 log.cc:824] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:45.397028 21617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec: No bootstrap required, opened a new log
I20260619 14:54:45.397024 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:54:45.404798 21620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:45.402255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48553 (local address 127.0.0.1:36311)
0619 14:54:45.402421 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:45.402426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:45.402893 (+ 467us) server_negotiation.cc:407] Connection header received
0619 14:54:45.403052 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:45.403056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:45.403100 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:45.403178 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:45.404139 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:45.404143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:45.404146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:45.404160 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:45.404193 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:45.404206 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:45.404302 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:45.404450 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 14:54:45.404493 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:54:45.407063 21617 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6d2ceb96a34d029236f18b4e5b43ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } }
I20260619 14:54:45.407397 21617 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:45.407543 21617 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc6d2ceb96a34d029236f18b4e5b43ec, State: Initialized, Role: FOLLOWER
I20260619 14:54:45.408061 21617 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6d2ceb96a34d029236f18b4e5b43ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } }
I20260619 14:54:45.408288 21617 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:45.408429 21617 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:45.408599 21617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:45.410861 21617 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6d2ceb96a34d029236f18b4e5b43ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } }
I20260619 14:54:45.411397 21617 leader_election.cc:302] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc6d2ceb96a34d029236f18b4e5b43ec; no voters:
I20260619 14:54:45.412014 21617 leader_election.cc:288] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:45.412140 21627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:45.412822 21627 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [term 1 LEADER]: Becoming Leader. State: Replica: cc6d2ceb96a34d029236f18b4e5b43ec, State: Running, Role: LEADER
I20260619 14:54:45.413365 21627 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6d2ceb96a34d029236f18b4e5b43ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } }
I20260619 14:54:45.413946 21617 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:45.415953 21629 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc6d2ceb96a34d029236f18b4e5b43ec. Latest consensus state: current_term: 1 leader_uuid: "cc6d2ceb96a34d029236f18b4e5b43ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6d2ceb96a34d029236f18b4e5b43ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } } }
I20260619 14:54:45.416327 21629 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:45.416860 21628 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc6d2ceb96a34d029236f18b4e5b43ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6d2ceb96a34d029236f18b4e5b43ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } } }
I20260619 14:54:45.417219 21628 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:45.420336 21634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:45.425823 21634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:45.433588 21634 catalog_manager.cc:1347] Generated new cluster ID: 1c966d29561f48d7b049add70d6f1d00
I20260619 14:54:45.433694 21634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:45.456466 21634 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:45.456893 21634 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:45.463083 21634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc6d2ceb96a34d029236f18b4e5b43ec: Generated new TSK 0
I20260619 14:54:45.463923 21634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:45.476992 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21550
W20260619 14:54:45.478148 21612 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46601
W20260619 14:54:45.478909 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21550: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:54:45.479066 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 21550
2026-06-19T14:54:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1863 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:45 dist-test-slave-p3ml krb5kdc[21649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:45 dist-test-slave-p3ml krb5kdc[21649](info): set up 2 sockets
Jun 19 14:54:45 dist-test-slave-p3ml krb5kdc[21649](info): commencing operation
krb5kdc: starting...
W20260619 14:54:46.575748 964 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:46 dist-test-slave-p3ml krb5kdc[21649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:54:46.818619 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:47.143981 21662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:47.144265 21662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:47.144429 21662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:47.151575 21662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:47.151676 21662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:47.151786 21662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:47.151894 21662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:47.151979 21662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:47.161492 21662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:47.162752 21662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:47.164072 21662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:47.175901 21670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:47.175974 21668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:47.176226 21667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:47.176720 21662 server_base.cc:1034] running on GCE node
I20260619 14:54:47.177508 21662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:47.178920 21662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:47.180148 21662 hybrid_clock.cc:648] HybridClock initialized: now 1781880887180113 us; error 51 us; skew 500 ppm
I20260619 14:54:47.180498 21662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:47 dist-test-slave-p3ml krb5kdc[21649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:47.188238 21662 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:54:47.189806 21662 webserver.cc:460] Webserver started at http://127.0.0.1:34559/ using document root <none> and password file <none>
I20260619 14:54:47.190519 21662 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:47.190673 21662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:47.190991 21662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:47.194068 21662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "4c76c22a4c4e4d478d85f043c612f955"
format_stamp: "Formatted at 2026-06-19 14:54:47 on dist-test-slave-p3ml"
server_key: "7e42e6f6da0549a2698fb01b67fbe7c0"
server_key_iv: "34be3613fc66cda3e751d17eb232231c"
server_key_version: "encryptionkey@0"
I20260619 14:54:47.194841 21662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "4c76c22a4c4e4d478d85f043c612f955"
format_stamp: "Formatted at 2026-06-19 14:54:47 on dist-test-slave-p3ml"
server_key: "7e42e6f6da0549a2698fb01b67fbe7c0"
server_key_iv: "34be3613fc66cda3e751d17eb232231c"
server_key_version: "encryptionkey@0"
I20260619 14:54:47.199936 21662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 14:54:47.203516 21677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:47.204800 21662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:54:47.205008 21662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "4c76c22a4c4e4d478d85f043c612f955"
format_stamp: "Formatted at 2026-06-19 14:54:47 on dist-test-slave-p3ml"
server_key: "7e42e6f6da0549a2698fb01b67fbe7c0"
server_key_iv: "34be3613fc66cda3e751d17eb232231c"
server_key_version: "encryptionkey@0"
I20260619 14:54:47.205216 21662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:47.219331 21662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:47.223125 21662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:47.223412 21662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:47.241864 21662 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33823
I20260619 14:54:47.241957 21728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33823 every 8 connection(s)
I20260619 14:54:47.243291 21662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:47.246989 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:47.252496 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 21662
I20260619 14:54:47.252718 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:47.253144 964 external_mini_cluster.cc:1428] Setting key 5468ccdcf02f638843a59a314dd1cdea
I20260619 14:54:47.258682 21729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955: Bootstrap starting.
I20260619 14:54:47.262420 21729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:47.263697 21729 log.cc:824] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:47.267365 21729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955: No bootstrap required, opened a new log
Jun 19 14:54:47 dist-test-slave-p3ml krb5kdc[21649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:54:47.275417 21729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c76c22a4c4e4d478d85f043c612f955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33823 } }
I20260619 14:54:47.275934 21729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:47.276072 21729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c76c22a4c4e4d478d85f043c612f955, State: Initialized, Role: FOLLOWER
I20260619 14:54:47.276662 21729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c76c22a4c4e4d478d85f043c612f955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33823 } }
I20260619 14:54:47.276901 21729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:47.277081 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:47.254938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35037 (local address 127.0.0.1:33823)
0619 14:54:47.255813 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:47.255824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:47.255863 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:54:47.257572 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:47.257594 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:47.257977 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:47.258313 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:47.260109 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:47.261518 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:47.263416 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:47.264345 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:47.264561 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:47.269267 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:47.269275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:47.269293 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:47.269327 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:47.272546 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:47.273301 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:47.273312 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:47.273318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:47.273491 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:47.273887 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:47.273892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:47.273897 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:47.274443 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:47.274765 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:47.275232 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 14:54:47.275710 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":381,"threads_started":1}
I20260619 14:54:47.277082 21729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:47.277314 21729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:47.279832 21729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c76c22a4c4e4d478d85f043c612f955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33823 } }
I20260619 14:54:47.280387 21729 leader_election.cc:302] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c76c22a4c4e4d478d85f043c612f955; no voters:
I20260619 14:54:47.280864 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:54:47.281061 21729 leader_election.cc:288] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:47.281163 21734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:47.282966 21729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:47.283736 21734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [term 1 LEADER]: Becoming Leader. State: Replica: 4c76c22a4c4e4d478d85f043c612f955, State: Running, Role: LEADER
I20260619 14:54:47.284451 21734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c76c22a4c4e4d478d85f043c612f955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33823 } }
I20260619 14:54:47.288935 21736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c76c22a4c4e4d478d85f043c612f955. Latest consensus state: current_term: 1 leader_uuid: "4c76c22a4c4e4d478d85f043c612f955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c76c22a4c4e4d478d85f043c612f955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33823 } } }
I20260619 14:54:47.289315 21736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:47.289927 21735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c76c22a4c4e4d478d85f043c612f955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c76c22a4c4e4d478d85f043c612f955" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33823 } } }
I20260619 14:54:47.290253 21735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:47.291560 21745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:47.297266 21745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:47.308892 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:47.287137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33245 (local address 127.0.0.1:33823)
0619 14:54:47.287361 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:47.287370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:47.287997 (+ 627us) server_negotiation.cc:407] Connection header received
0619 14:54:47.288298 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:47.288303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:47.288388 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:47.288466 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:47.290705 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:47.291953 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:47.294051 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:47.294885 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:47.295043 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:47.299231 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:47.299241 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:47.299250 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:47.299308 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:47.304299 (+ 4991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:47.305485 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:47.305494 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:47.305500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:47.305690 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:47.306267 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:47.306275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:47.306284 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:47.306846 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:47.307158 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:47.307980 (+ 822us) server_negotiation.cc:299] Negotiation successful
0619 14:54:47.308372 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:54:47.319187 21745 catalog_manager.cc:1347] Generated new cluster ID: 16a2ccae92c94cd79eb1a2f8d26f3e45
I20260619 14:54:47.319350 21745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:47.331120 21745 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:47.332617 21745 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:47.342394 21745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c76c22a4c4e4d478d85f043c612f955: Generated new TSK 0
I20260619 14:54:47.343396 21745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:47.350971 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21662
I20260619 14:54:47.352172 21724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:47.443302 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 21662
2026-06-19T14:54:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:47 dist-test-slave-p3ml krb5kdc[21761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:47 dist-test-slave-p3ml krb5kdc[21761](info): set up 2 sockets
Jun 19 14:54:47 dist-test-slave-p3ml krb5kdc[21761](info): commencing operation
krb5kdc: starting...
W20260619 14:54:48.548027 964 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:48 dist-test-slave-p3ml krb5kdc[21761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:54:48.785919 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:49.120843 21774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:49.121222 21774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:49.121399 21774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:49.128885 21774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:49.129050 21774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:49.129161 21774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:49.129277 21774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:49.129369 21774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:49.139405 21774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:49.140753 21774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:49.142163 21774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:49.154193 21779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:49.154088 21780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:49.154161 21782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:49.155419 21774 server_base.cc:1034] running on GCE node
I20260619 14:54:49.156257 21774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:49.157574 21774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:49.158797 21774 hybrid_clock.cc:648] HybridClock initialized: now 1781880889158755 us; error 35 us; skew 500 ppm
I20260619 14:54:49.159209 21774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:49 dist-test-slave-p3ml krb5kdc[21761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:49.162403 21774 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:54:49.163836 21774 webserver.cc:460] Webserver started at http://127.0.0.1:33531/ using document root <none> and password file <none>
I20260619 14:54:49.164487 21774 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:49.164636 21774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:49.164942 21774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:49.167868 21774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "7f38a655ef1347d5b9db8cea60e237cf"
format_stamp: "Formatted at 2026-06-19 14:54:49 on dist-test-slave-p3ml"
server_key: "0be0446af7f8f7556097664b95d118a4"
server_key_iv: "80fbc43db14e40d375d1d953a12c6098"
server_key_version: "encryptionkey@0"
I20260619 14:54:49.168613 21774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "7f38a655ef1347d5b9db8cea60e237cf"
format_stamp: "Formatted at 2026-06-19 14:54:49 on dist-test-slave-p3ml"
server_key: "0be0446af7f8f7556097664b95d118a4"
server_key_iv: "80fbc43db14e40d375d1d953a12c6098"
server_key_version: "encryptionkey@0"
I20260619 14:54:49.173465 21774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 14:54:49.176695 21789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:49.177901 21774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:54:49.178079 21774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "7f38a655ef1347d5b9db8cea60e237cf"
format_stamp: "Formatted at 2026-06-19 14:54:49 on dist-test-slave-p3ml"
server_key: "0be0446af7f8f7556097664b95d118a4"
server_key_iv: "80fbc43db14e40d375d1d953a12c6098"
server_key_version: "encryptionkey@0"
I20260619 14:54:49.178277 21774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:49.193585 21774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:49.197326 21774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:49.197603 21774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:49.215854 21774 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46377
I20260619 14:54:49.215935 21840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46377 every 8 connection(s)
I20260619 14:54:49.217238 21774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:49.217674 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 21774
I20260619 14:54:49.217885 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:49.218492 964 external_mini_cluster.cc:1428] Setting key 21ca6e40ddd2dd7f4abd4c61bffb328e
I20260619 14:54:49.221525 21841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:49.231683 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:49.220295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59215 (local address 127.0.0.1:46377)
0619 14:54:49.221046 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:49.221058 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:49.221097 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:54:49.222781 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:49.222803 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:49.223163 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:49.223544 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:49.224079 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.225406 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.227392 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.228104 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.228240 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:49.229382 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:49.229389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:49.229401 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:49.229932 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:49.230032 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:49.230057 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:49.230165 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:49.230384 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 14:54:49.230653 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":348,"threads_started":1}
I20260619 14:54:49.233863 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf: Bootstrap starting.
I20260619 14:54:49.235421 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:54:49.237473 21841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:49.238762 21841 log.cc:824] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:49.242162 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf: No bootstrap required, opened a new log
I20260619 14:54:49.247195 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:49.240658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47297 (local address 127.0.0.1:46377)
0619 14:54:49.240866 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:49.240875 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:49.241359 (+ 484us) server_negotiation.cc:407] Connection header received
0619 14:54:49.241536 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:49.241540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:49.241604 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:49.241679 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:49.242384 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.243095 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.244900 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.245367 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.245456 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:49.246529 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:49.246533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:49.246535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:49.246550 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:49.246592 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:49.246608 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:49.246669 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:49.246816 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 14:54:49.246879 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:54:49.251086 21841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f38a655ef1347d5b9db8cea60e237cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46377 } }
I20260619 14:54:49.251430 21841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:49.251595 21841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f38a655ef1347d5b9db8cea60e237cf, State: Initialized, Role: FOLLOWER
I20260619 14:54:49.252264 21841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f38a655ef1347d5b9db8cea60e237cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46377 } }
I20260619 14:54:49.252570 21841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:49.252753 21841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:49.252924 21841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:49.254946 21841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f38a655ef1347d5b9db8cea60e237cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46377 } }
I20260619 14:54:49.255405 21841 leader_election.cc:302] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f38a655ef1347d5b9db8cea60e237cf; no voters:
I20260619 14:54:49.255954 21841 leader_election.cc:288] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:49.256094 21851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:49.257356 21851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [term 1 LEADER]: Becoming Leader. State: Replica: 7f38a655ef1347d5b9db8cea60e237cf, State: Running, Role: LEADER
I20260619 14:54:49.257928 21851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f38a655ef1347d5b9db8cea60e237cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46377 } }
I20260619 14:54:49.258178 21841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:49.261835 21853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f38a655ef1347d5b9db8cea60e237cf. Latest consensus state: current_term: 1 leader_uuid: "7f38a655ef1347d5b9db8cea60e237cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f38a655ef1347d5b9db8cea60e237cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46377 } } }
I20260619 14:54:49.261899 21852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f38a655ef1347d5b9db8cea60e237cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f38a655ef1347d5b9db8cea60e237cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46377 } } }
I20260619 14:54:49.262236 21853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:49.262318 21852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:49.265280 21858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:49.270056 21858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:49.278487 21858 catalog_manager.cc:1347] Generated new cluster ID: 6ad7c925b8524a51a4260b1a7070755c
I20260619 14:54:49.278589 21858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:49.300091 21858 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:49.301914 21858 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:49.313488 21858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f38a655ef1347d5b9db8cea60e237cf: Generated new TSK 0
I20260619 14:54:49.314451 21858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:49.328915 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21774
W20260619 14:54:49.329934 21836 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59215
W20260619 14:54:49.330618 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21774: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:54:49.330750 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 21774
2026-06-19T14:54:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1886 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:49 dist-test-slave-p3ml krb5kdc[21872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:49 dist-test-slave-p3ml krb5kdc[21872](info): set up 2 sockets
Jun 19 14:54:49 dist-test-slave-p3ml krb5kdc[21872](info): commencing operation
krb5kdc: starting...
W20260619 14:54:50.419646 964 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:50 dist-test-slave-p3ml krb5kdc[21872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:54:50.668151 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:50.989063 21885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:50.989341 21885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:50.989547 21885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:50.996654 21885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:50.996802 21885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:50.996889 21885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:50.997009 21885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:50.997094 21885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:51.006821 21885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:51.008138 21885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:51.009418 21885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:51.021586 21891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:51.021618 21890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:51.021814 21893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:51.022298 21885 server_base.cc:1034] running on GCE node
I20260619 14:54:51.023074 21885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:51.024502 21885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:51.025763 21885 hybrid_clock.cc:648] HybridClock initialized: now 1781880891025728 us; error 37 us; skew 500 ppm
I20260619 14:54:51.026150 21885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:51 dist-test-slave-p3ml krb5kdc[21872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:51.029280 21885 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:54:51.030687 21885 webserver.cc:460] Webserver started at http://127.0.0.1:36591/ using document root <none> and password file <none>
I20260619 14:54:51.031355 21885 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:51.031505 21885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:51.031831 21885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:51.034751 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "626a9efd09124e23b7fa3a2de5adde19"
format_stamp: "Formatted at 2026-06-19 14:54:51 on dist-test-slave-p3ml"
server_key: "0ea56bc0359fdbbdf2ff4c678c2f1275"
server_key_iv: "6085e76c8a8e749734ad272898b21943"
server_key_version: "encryptionkey@0"
I20260619 14:54:51.035575 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "626a9efd09124e23b7fa3a2de5adde19"
format_stamp: "Formatted at 2026-06-19 14:54:51 on dist-test-slave-p3ml"
server_key: "0ea56bc0359fdbbdf2ff4c678c2f1275"
server_key_iv: "6085e76c8a8e749734ad272898b21943"
server_key_version: "encryptionkey@0"
I20260619 14:54:51.040539 21885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 14:54:51.043838 21900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:51.045070 21885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 14:54:51.045265 21885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "626a9efd09124e23b7fa3a2de5adde19"
format_stamp: "Formatted at 2026-06-19 14:54:51 on dist-test-slave-p3ml"
server_key: "0ea56bc0359fdbbdf2ff4c678c2f1275"
server_key_iv: "6085e76c8a8e749734ad272898b21943"
server_key_version: "encryptionkey@0"
I20260619 14:54:51.045454 21885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:51.051167 21885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:51.054883 21885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:51.055195 21885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:51.073776 21885 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36629
I20260619 14:54:51.073841 21951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36629 every 8 connection(s)
I20260619 14:54:51.075080 21885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:51.078696 21952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:51.081823 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 21885
I20260619 14:54:51.082023 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:51.082465 964 external_mini_cluster.cc:1428] Setting key 248f41ea1fb5f197d8d5664da605385f
I20260619 14:54:51.091076 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:51.084311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33299 (local address 127.0.0.1:36629)
0619 14:54:51.085101 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:51.085116 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:51.085169 (+ 53us) server_negotiation.cc:407] Connection header received
0619 14:54:51.087329 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:51.087363 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:51.087841 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:51.088274 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:51.089249 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:51.089256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:51.089272 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:51.089297 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:51.089364 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:51.089388 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:51.089613 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:51.089757 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 14:54:51.089935 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":338,"threads_started":1}
I20260619 14:54:51.092065 21952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19: Bootstrap starting.
I20260619 14:54:51.094656 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:54:51.095425 21952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:51.096465 21952 log.cc:824] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:51.099545 21952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19: No bootstrap required, opened a new log
I20260619 14:54:51.101317 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:51.098969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43721 (local address 127.0.0.1:36629)
0619 14:54:51.099138 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:51.099143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:51.099469 (+ 326us) server_negotiation.cc:407] Connection header received
0619 14:54:51.099670 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:51.099675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:51.099729 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:51.099835 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:51.100710 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:51.100715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:51.100720 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:51.100736 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:51.100780 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:51.100801 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:51.100922 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:51.100969 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 14:54:51.101025 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:54:51.107045 21952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626a9efd09124e23b7fa3a2de5adde19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36629 } }
I20260619 14:54:51.107415 21952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:51.107553 21952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626a9efd09124e23b7fa3a2de5adde19, State: Initialized, Role: FOLLOWER
I20260619 14:54:51.108157 21952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626a9efd09124e23b7fa3a2de5adde19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36629 } }
I20260619 14:54:51.108393 21952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:51.108541 21952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:51.108700 21952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:51.111088 21952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626a9efd09124e23b7fa3a2de5adde19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36629 } }
I20260619 14:54:51.111594 21952 leader_election.cc:302] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 626a9efd09124e23b7fa3a2de5adde19; no voters:
I20260619 14:54:51.112130 21952 leader_election.cc:288] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:51.112316 21962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:51.113070 21962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [term 1 LEADER]: Becoming Leader. State: Replica: 626a9efd09124e23b7fa3a2de5adde19, State: Running, Role: LEADER
I20260619 14:54:51.113709 21962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626a9efd09124e23b7fa3a2de5adde19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36629 } }
I20260619 14:54:51.114229 21952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:51.116791 21964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 626a9efd09124e23b7fa3a2de5adde19. Latest consensus state: current_term: 1 leader_uuid: "626a9efd09124e23b7fa3a2de5adde19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626a9efd09124e23b7fa3a2de5adde19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36629 } } }
I20260619 14:54:51.117259 21964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:51.118058 21963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "626a9efd09124e23b7fa3a2de5adde19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626a9efd09124e23b7fa3a2de5adde19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36629 } } }
I20260619 14:54:51.118436 21963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:51.123023 21971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:51.127840 21971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:51.136178 21971 catalog_manager.cc:1347] Generated new cluster ID: f934e57de7d14aa7ac4e505b3fd9feb0
I20260619 14:54:51.136294 21971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:51.153790 21971 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:51.154371 21971 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:51.170194 21971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 626a9efd09124e23b7fa3a2de5adde19: Generated new TSK 0
I20260619 14:54:51.171337 21971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:51.181578 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21885
W20260619 14:54:51.182747 21947 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33299
W20260619 14:54:51.183511 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21885: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:54:51.183681 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 21885
2026-06-19T14:54:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1853 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:51 dist-test-slave-p3ml krb5kdc[21984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:51 dist-test-slave-p3ml krb5kdc[21984](info): set up 2 sockets
Jun 19 14:54:51 dist-test-slave-p3ml krb5kdc[21984](info): commencing operation
krb5kdc: starting...
W20260619 14:54:52.287871 964 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:52 dist-test-slave-p3ml krb5kdc[21984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:54:52.535923 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:52.858754 21997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:52.859095 21997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:52.859275 21997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:52.866493 21997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:52.866636 21997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:52.866766 21997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:52.866878 21997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:52.866966 21997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:52.876729 21997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:52.877997 21997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:52.879268 21997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:52.890728 22003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:52.890789 22005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:52.890812 22002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:52.892047 21997 server_base.cc:1034] running on GCE node
I20260619 14:54:52.892813 21997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:52.894161 21997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:52.895399 21997 hybrid_clock.cc:648] HybridClock initialized: now 1781880892895355 us; error 33 us; skew 500 ppm
I20260619 14:54:52.895807 21997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:52 dist-test-slave-p3ml krb5kdc[21984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:52.902065 21997 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:54:52.903707 21997 webserver.cc:460] Webserver started at http://127.0.0.1:33263/ using document root <none> and password file <none>
I20260619 14:54:52.904389 21997 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:52.904546 21997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:52.904863 21997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:52.907884 21997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "4ae757cd57364d7fbc79a8b7dbd6c16c"
format_stamp: "Formatted at 2026-06-19 14:54:52 on dist-test-slave-p3ml"
server_key: "d9ca6d7467eeddaad0a74900309d152f"
server_key_iv: "f1936bbe29b05a741a5eaa3691df32ce"
server_key_version: "encryptionkey@0"
I20260619 14:54:52.908681 21997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "4ae757cd57364d7fbc79a8b7dbd6c16c"
format_stamp: "Formatted at 2026-06-19 14:54:52 on dist-test-slave-p3ml"
server_key: "d9ca6d7467eeddaad0a74900309d152f"
server_key_iv: "f1936bbe29b05a741a5eaa3691df32ce"
server_key_version: "encryptionkey@0"
I20260619 14:54:52.913704 21997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260619 14:54:52.917140 22012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:52.918535 21997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:54:52.918735 21997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "4ae757cd57364d7fbc79a8b7dbd6c16c"
format_stamp: "Formatted at 2026-06-19 14:54:52 on dist-test-slave-p3ml"
server_key: "d9ca6d7467eeddaad0a74900309d152f"
server_key_iv: "f1936bbe29b05a741a5eaa3691df32ce"
server_key_version: "encryptionkey@0"
I20260619 14:54:52.918932 21997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:52.937670 21997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:52.941847 21997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:52.942133 21997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:52.960597 21997 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43263
I20260619 14:54:52.960686 22063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43263 every 8 connection(s)
I20260619 14:54:52.961926 21997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:52.965555 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:52.968875 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 21997
I20260619 14:54:52.969074 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:52.969599 964 external_mini_cluster.cc:1428] Setting key f3e0475e4dc4f780fa8d632a1ab73f05
I20260619 14:54:52.977208 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c: Bootstrap starting.
I20260619 14:54:52.980295 22064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:52.981333 22064 log.cc:824] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:54:52 dist-test-slave-p3ml krb5kdc[21984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:54:52.984831 22064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c: No bootstrap required, opened a new log
I20260619 14:54:52.992075 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:52.971479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:36117 (local address 127.0.0.1:43263)
0619 14:54:52.972158 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:52.972169 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:52.972219 (+ 50us) server_negotiation.cc:407] Connection header received
0619 14:54:52.973845 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:52.973865 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:52.974259 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:52.974647 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:52.976218 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:52.977537 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:52.978821 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:52.979560 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:52.979748 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:52.984461 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:52.984469 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:52.984483 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:52.984517 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:52.987823 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:52.988620 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:52.988626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:52.988632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:52.988806 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:52.989283 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:52.989292 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:52.989295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:52.989832 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:52.990207 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:52.990660 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 14:54:52.990970 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":294,"threads_started":1}
I20260619 14:54:52.995016 22064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae757cd57364d7fbc79a8b7dbd6c16c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } }
I20260619 14:54:52.995604 22064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:52.995714 22064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae757cd57364d7fbc79a8b7dbd6c16c, State: Initialized, Role: FOLLOWER
I20260619 14:54:52.996507 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:54:52.996446 22064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae757cd57364d7fbc79a8b7dbd6c16c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } }
I20260619 14:54:52.996819 22064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:52.997004 22064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:52.997212 22064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:53.000136 22064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae757cd57364d7fbc79a8b7dbd6c16c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } }
I20260619 14:54:53.000798 22064 leader_election.cc:302] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ae757cd57364d7fbc79a8b7dbd6c16c; no voters:
I20260619 14:54:53.001632 22064 leader_election.cc:288] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:53.001751 22074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:53.002430 22074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [term 1 LEADER]: Becoming Leader. State: Replica: 4ae757cd57364d7fbc79a8b7dbd6c16c, State: Running, Role: LEADER
I20260619 14:54:53.003054 22074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae757cd57364d7fbc79a8b7dbd6c16c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } }
I20260619 14:54:53.005690 22064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:53.005998 22075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ae757cd57364d7fbc79a8b7dbd6c16c. Latest consensus state: current_term: 1 leader_uuid: "4ae757cd57364d7fbc79a8b7dbd6c16c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae757cd57364d7fbc79a8b7dbd6c16c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } } }
I20260619 14:54:53.005987 22076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ae757cd57364d7fbc79a8b7dbd6c16c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae757cd57364d7fbc79a8b7dbd6c16c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } } }
I20260619 14:54:53.006467 22076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:53.006471 22075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:53.007308 22078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:53.013975 22078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:53.028910 22067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:53.000933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:43035 (local address 127.0.0.1:43263)
0619 14:54:53.001095 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:53.001099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:53.001802 (+ 703us) server_negotiation.cc:407] Connection header received
0619 14:54:53.002071 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:53.002082 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:53.002221 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:53.002394 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:53.005070 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:53.006498 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:53.009308 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:53.010399 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:53.010564 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:53.016147 (+ 5583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:53.016159 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:53.016169 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:53.016234 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:53.022899 (+ 6665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:53.024552 (+ 1653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:53.024574 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:53.024581 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:53.024790 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:53.025709 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:53.025719 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:53.025726 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:53.026395 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:53.026839 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:53.027926 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0619 14:54:53.028219 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:54:53.034613 22078 catalog_manager.cc:1347] Generated new cluster ID: f9c3f26e944441a08f9fe7c8460f2b7e
I20260619 14:54:53.034746 22078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:53.055307 22078 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:53.056602 22078 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:53.066201 22078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ae757cd57364d7fbc79a8b7dbd6c16c: Generated new TSK 0
I20260619 14:54:53.067046 22078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:53.077811 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 21997
I20260619 14:54:53.078951 22059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:53.171272 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 21997
W20260619 14:54:53.188512 21982 connection.cc:535] client connection to 127.0.0.1:43263 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43263): Connection reset by peer (error 104)
2026-06-19T14:54:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:53 dist-test-slave-p3ml krb5kdc[22096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:53 dist-test-slave-p3ml krb5kdc[22096](info): set up 2 sockets
Jun 19 14:54:53 dist-test-slave-p3ml krb5kdc[22096](info): commencing operation
krb5kdc: starting...
W20260619 14:54:54.275786 964 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:54 dist-test-slave-p3ml krb5kdc[22096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:54:54.530823 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:54.853313 22109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:54.853672 22109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:54.853863 22109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:54.861339 22109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:54.861479 22109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:54.861603 22109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:54.861729 22109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:54.861809 22109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:54.871956 22109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:54.873265 22109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:54.874612 22109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:54.886591 22117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:54.886672 22114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:54.886554 22115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:54.887599 22109 server_base.cc:1034] running on GCE node
I20260619 14:54:54.888465 22109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:54.889933 22109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:54.891139 22109 hybrid_clock.cc:648] HybridClock initialized: now 1781880894891069 us; error 59 us; skew 500 ppm
I20260619 14:54:54.891559 22109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:54 dist-test-slave-p3ml krb5kdc[22096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:54.894609 22109 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:54:54.896123 22109 webserver.cc:460] Webserver started at http://127.0.0.1:40197/ using document root <none> and password file <none>
I20260619 14:54:54.896831 22109 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:54.896973 22109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:54.897282 22109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:54.900287 22109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "e84014bbdaf84fa3b5aaaa5032c04f57"
format_stamp: "Formatted at 2026-06-19 14:54:54 on dist-test-slave-p3ml"
server_key: "3fb287de252505864a9572ed2a6f9ece"
server_key_iv: "a70906268499d97cb523a1d5ae3f9b4e"
server_key_version: "encryptionkey@0"
I20260619 14:54:54.901062 22109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "e84014bbdaf84fa3b5aaaa5032c04f57"
format_stamp: "Formatted at 2026-06-19 14:54:54 on dist-test-slave-p3ml"
server_key: "3fb287de252505864a9572ed2a6f9ece"
server_key_iv: "a70906268499d97cb523a1d5ae3f9b4e"
server_key_version: "encryptionkey@0"
I20260619 14:54:54.906258 22109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:54:54.909669 22124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:54.911005 22109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:54:54.911192 22109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "e84014bbdaf84fa3b5aaaa5032c04f57"
format_stamp: "Formatted at 2026-06-19 14:54:54 on dist-test-slave-p3ml"
server_key: "3fb287de252505864a9572ed2a6f9ece"
server_key_iv: "a70906268499d97cb523a1d5ae3f9b4e"
server_key_version: "encryptionkey@0"
I20260619 14:54:54.911386 22109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:54.944072 22109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:54.948849 22109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:54.949261 22109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:54.969952 22109 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46173
I20260619 14:54:54.969988 22175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46173 every 8 connection(s)
I20260619 14:54:54.971460 22109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:54.973016 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 22109
I20260619 14:54:54.973217 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:54.973672 964 external_mini_cluster.cc:1428] Setting key 1598adf40f0f2fac60bf58c70045b4e4
I20260619 14:54:54.975982 22176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:54.988101 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:54.975735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:36111 (local address 127.0.0.1:46173)
0619 14:54:54.976575 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:54.976587 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:54.976630 (+ 43us) server_negotiation.cc:407] Connection header received
0619 14:54:54.978419 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:54.978442 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:54.978916 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:54.979338 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:54.980018 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:54.981437 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:54.983536 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:54.984248 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:54.984380 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:54.985726 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:54.985733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:54.985747 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:54.986325 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:54.986399 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:54.986430 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:54.986549 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:54.986837 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 14:54:54.987054 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":364,"threads_started":1}
I20260619 14:54:54.989215 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57: Bootstrap starting.
I20260619 14:54:54.992256 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:54:54.993160 22176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:54.994161 22176 log.cc:824] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:54.997359 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57: No bootstrap required, opened a new log
I20260619 14:54:55.003775 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:54.996581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:37403 (local address 127.0.0.1:46173)
0619 14:54:54.996784 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:54.996794 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:54.997318 (+ 524us) server_negotiation.cc:407] Connection header received
0619 14:54:54.997535 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:54.997539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:54.997599 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:54.997679 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:54.998388 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:54.999147 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:55.001151 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:55.001656 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:55.001758 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:55.002953 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:55.002957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:55.002960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:55.002976 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:55.003012 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:55.003026 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:55.003113 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:55.003334 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 14:54:55.003375 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:54:55.007113 22176 raft_consensus.cc:348] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84014bbdaf84fa3b5aaaa5032c04f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } }
I20260619 14:54:55.007573 22176 raft_consensus.cc:374] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:55.007730 22176 raft_consensus.cc:729] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84014bbdaf84fa3b5aaaa5032c04f57, State: Initialized, Role: FOLLOWER
I20260619 14:54:55.008390 22176 consensus_queue.cc:260] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84014bbdaf84fa3b5aaaa5032c04f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } }
I20260619 14:54:55.008621 22176 raft_consensus.cc:388] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:55.008764 22176 raft_consensus.cc:482] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:55.008942 22176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:55.011111 22176 raft_consensus.cc:504] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84014bbdaf84fa3b5aaaa5032c04f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } }
I20260619 14:54:55.011693 22176 leader_election.cc:302] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e84014bbdaf84fa3b5aaaa5032c04f57; no voters:
I20260619 14:54:55.012245 22176 leader_election.cc:288] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:55.012414 22186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:55.013078 22186 raft_consensus.cc:686] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [term 1 LEADER]: Becoming Leader. State: Replica: e84014bbdaf84fa3b5aaaa5032c04f57, State: Running, Role: LEADER
I20260619 14:54:55.013592 22186 consensus_queue.cc:237] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84014bbdaf84fa3b5aaaa5032c04f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } }
I20260619 14:54:55.014366 22176 sys_catalog.cc:564] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:55.016816 22188 sys_catalog.cc:455] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e84014bbdaf84fa3b5aaaa5032c04f57. Latest consensus state: current_term: 1 leader_uuid: "e84014bbdaf84fa3b5aaaa5032c04f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84014bbdaf84fa3b5aaaa5032c04f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } } }
I20260619 14:54:55.016722 22187 sys_catalog.cc:455] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e84014bbdaf84fa3b5aaaa5032c04f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84014bbdaf84fa3b5aaaa5032c04f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } } }
I20260619 14:54:55.017107 22188 sys_catalog.cc:458] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:55.017107 22187 sys_catalog.cc:458] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:55.020531 22192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:55.025664 22192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:55.035336 22192 catalog_manager.cc:1347] Generated new cluster ID: f48a812b02244a728ecad7412667bd20
I20260619 14:54:55.035467 22192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:55.056861 22192 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:55.058213 22192 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:55.073529 22192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e84014bbdaf84fa3b5aaaa5032c04f57: Generated new TSK 0
I20260619 14:54:55.074411 22192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:55.081792 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22109
W20260619 14:54:55.083295 22171 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.56:36111
W20260619 14:54:55.084077 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22109: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:54:55.084231 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 22109
W20260619 14:54:55.101190 22094 connection.cc:535] client connection to 127.0.0.1:46173 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46173): Connection reset by peer (error 104)
2026-06-19T14:54:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:55 dist-test-slave-p3ml krb5kdc[22207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:55 dist-test-slave-p3ml krb5kdc[22207](info): set up 2 sockets
Jun 19 14:54:55 dist-test-slave-p3ml krb5kdc[22207](info): commencing operation
krb5kdc: starting...
W20260619 14:54:56.195812 964 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:56 dist-test-slave-p3ml krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:54:56.435734 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:56.769835 22220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:56.770107 22220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:56.770231 22220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:56.777453 22220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:56.777582 22220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:56.777688 22220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:56.777804 22220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:56.777897 22220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:56.787384 22220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:56.788630 22220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:56.789912 22220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:56.802115 22225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:56.802198 22228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:56.802593 22226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:56.802848 22220 server_base.cc:1034] running on GCE node
I20260619 14:54:56.803650 22220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:56.804913 22220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:56.806110 22220 hybrid_clock.cc:648] HybridClock initialized: now 1781880896806082 us; error 29 us; skew 500 ppm
I20260619 14:54:56.806433 22220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:56 dist-test-slave-p3ml krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:56.809506 22220 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:54:56.810971 22220 webserver.cc:460] Webserver started at http://127.0.0.1:40951/ using document root <none> and password file <none>
I20260619 14:54:56.811730 22220 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:56.811862 22220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:56.812166 22220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:56.814993 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "8c4ac0b096aa44e897dffd348cb993fe"
format_stamp: "Formatted at 2026-06-19 14:54:56 on dist-test-slave-p3ml"
server_key: "b55e09b53de13a6da7e77365ee6e8564"
server_key_iv: "9a6ae69d19dd1157cb91e21d289c419c"
server_key_version: "encryptionkey@0"
I20260619 14:54:56.815810 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "8c4ac0b096aa44e897dffd348cb993fe"
format_stamp: "Formatted at 2026-06-19 14:54:56 on dist-test-slave-p3ml"
server_key: "b55e09b53de13a6da7e77365ee6e8564"
server_key_iv: "9a6ae69d19dd1157cb91e21d289c419c"
server_key_version: "encryptionkey@0"
I20260619 14:54:56.820670 22220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:54:56.823940 22235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:56.825196 22220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 14:54:56.825384 22220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "8c4ac0b096aa44e897dffd348cb993fe"
format_stamp: "Formatted at 2026-06-19 14:54:56 on dist-test-slave-p3ml"
server_key: "b55e09b53de13a6da7e77365ee6e8564"
server_key_iv: "9a6ae69d19dd1157cb91e21d289c419c"
server_key_version: "encryptionkey@0"
I20260619 14:54:56.825579 22220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:56.831244 22220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:56.834986 22220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:56.835281 22220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:56.853662 22220 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37769
I20260619 14:54:56.853708 22286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37769 every 8 connection(s)
I20260619 14:54:56.855110 22220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:56.858915 22287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:56.859208 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 22220
I20260619 14:54:56.859388 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:56.859896 964 external_mini_cluster.cc:1428] Setting key 9f74239f17cb10478dcd594fc444af4e
I20260619 14:54:56.867298 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:56.861615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:33317 (local address 127.0.0.1:37769)
0619 14:54:56.862435 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:56.862446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:56.862483 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:54:56.864003 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:56.864022 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:56.864337 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:56.864727 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:56.865732 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:56.865738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:56.865750 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:56.865762 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:56.865820 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:56.865837 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:56.866075 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:56.866350 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 14:54:56.866496 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":331,"threads_started":1}
I20260619 14:54:56.870831 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:54:56.872512 22287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe: Bootstrap starting.
I20260619 14:54:56.875638 22287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:56.876776 22287 log.cc:824] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:56.877507 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:56.875111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:35337 (local address 127.0.0.1:37769)
0619 14:54:56.875302 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:56.875308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:56.875601 (+ 293us) server_negotiation.cc:407] Connection header received
0619 14:54:56.875790 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:56.875799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:56.875840 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:56.875952 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:56.876850 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:56.876862 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:56.876865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:54:56.876879 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:56.876934 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:54:56.876953 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:56.877077 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:56.877200 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 14:54:56.877255 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:54:56.879423 22287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe: No bootstrap required, opened a new log
I20260619 14:54:56.886310 22287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4ac0b096aa44e897dffd348cb993fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37769 } }
I20260619 14:54:56.886654 22287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:56.886776 22287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4ac0b096aa44e897dffd348cb993fe, State: Initialized, Role: FOLLOWER
I20260619 14:54:56.887344 22287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4ac0b096aa44e897dffd348cb993fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37769 } }
I20260619 14:54:56.887670 22287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:56.887830 22287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:56.888010 22287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:56.890090 22287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4ac0b096aa44e897dffd348cb993fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37769 } }
I20260619 14:54:56.890568 22287 leader_election.cc:302] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c4ac0b096aa44e897dffd348cb993fe; no voters:
I20260619 14:54:56.891112 22287 leader_election.cc:288] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:56.891247 22297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:56.892007 22297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [term 1 LEADER]: Becoming Leader. State: Replica: 8c4ac0b096aa44e897dffd348cb993fe, State: Running, Role: LEADER
I20260619 14:54:56.892606 22297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4ac0b096aa44e897dffd348cb993fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37769 } }
I20260619 14:54:56.893203 22287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:56.895673 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c4ac0b096aa44e897dffd348cb993fe. Latest consensus state: current_term: 1 leader_uuid: "8c4ac0b096aa44e897dffd348cb993fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4ac0b096aa44e897dffd348cb993fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37769 } } }
I20260619 14:54:56.896140 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:56.907833 22298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c4ac0b096aa44e897dffd348cb993fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4ac0b096aa44e897dffd348cb993fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37769 } } }
I20260619 14:54:56.908278 22298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:56.908818 22305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:56.913609 22305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:56.921947 22305 catalog_manager.cc:1347] Generated new cluster ID: dc1a14ed143e47dc9710fd20798dfc67
I20260619 14:54:56.922075 22305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:56.954458 22305 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:56.954963 22305 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:56.963052 22305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c4ac0b096aa44e897dffd348cb993fe: Generated new TSK 0
I20260619 14:54:56.964003 22305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:56.977507 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22220
W20260619 14:54:56.978688 22282 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.56:33317
W20260619 14:54:56.979352 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22220: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:54:56.979532 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 22220
2026-06-19T14:54:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1895 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:57 dist-test-slave-p3ml krb5kdc[22318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:57 dist-test-slave-p3ml krb5kdc[22318](info): set up 2 sockets
Jun 19 14:54:57 dist-test-slave-p3ml krb5kdc[22318](info): commencing operation
krb5kdc: starting...
W20260619 14:54:58.092002 964 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:54:58 dist-test-slave-p3ml krb5kdc[22318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:54:58.330446 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:58.648802 22331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:58.649137 22331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:58.649307 22331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:58.656529 22331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:58.656675 22331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:58.656791 22331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:58.656903 22331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:58.656980 22331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:58.666582 22331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:54:58.667894 22331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:58.669173 22331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:58.680898 22336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:58.680999 22337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:58.680977 22339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:58.681565 22331 server_base.cc:1034] running on GCE node
I20260619 14:54:58.682229 22331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:58.683535 22331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:58.684767 22331 hybrid_clock.cc:648] HybridClock initialized: now 1781880898684723 us; error 51 us; skew 500 ppm
I20260619 14:54:58.685112 22331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:54:58 dist-test-slave-p3ml krb5kdc[22318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:58.692492 22331 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:54:58.693866 22331 webserver.cc:460] Webserver started at http://127.0.0.1:44361/ using document root <none> and password file <none>
I20260619 14:54:58.694520 22331 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:58.694669 22331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:58.694968 22331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:58.697875 22331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "3f87a839395943849e0adc4ed1f0a551"
format_stamp: "Formatted at 2026-06-19 14:54:58 on dist-test-slave-p3ml"
server_key: "41a7ac741117facdb69266d31c14968e"
server_key_iv: "d25689b8db0a077eb210964931da14c8"
server_key_version: "encryptionkey@0"
I20260619 14:54:58.698604 22331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "3f87a839395943849e0adc4ed1f0a551"
format_stamp: "Formatted at 2026-06-19 14:54:58 on dist-test-slave-p3ml"
server_key: "41a7ac741117facdb69266d31c14968e"
server_key_iv: "d25689b8db0a077eb210964931da14c8"
server_key_version: "encryptionkey@0"
I20260619 14:54:58.703508 22331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 14:54:58.706630 22346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:58.707906 22331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:54:58.708082 22331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "3f87a839395943849e0adc4ed1f0a551"
format_stamp: "Formatted at 2026-06-19 14:54:58 on dist-test-slave-p3ml"
server_key: "41a7ac741117facdb69266d31c14968e"
server_key_iv: "d25689b8db0a077eb210964931da14c8"
server_key_version: "encryptionkey@0"
I20260619 14:54:58.708274 22331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:58.737219 22331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:58.740957 22331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:58.741252 22331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:58.759490 22331 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46307
I20260619 14:54:58.759574 22397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46307 every 8 connection(s)
I20260619 14:54:58.760843 22331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:58.763790 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 22331
I20260619 14:54:58.763965 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:54:58.764441 964 external_mini_cluster.cc:1428] Setting key 6b8d865e3b3dd0e79cb84cf9363ebca4
I20260619 14:54:58.764577 22398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:58.778641 22398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551: Bootstrap starting.
Jun 19 14:54:58 dist-test-slave-p3ml krb5kdc[22318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:54:58.782407 22398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:58.783778 22398 log.cc:824] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:58.787108 22398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551: No bootstrap required, opened a new log
I20260619 14:54:58.788950 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:58.766244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:52217 (local address 127.0.0.1:46307)
0619 14:54:58.767106 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:58.767125 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:58.767174 (+ 49us) server_negotiation.cc:407] Connection header received
0619 14:54:58.769443 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:58.769470 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:58.769988 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:58.770443 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:58.771755 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:58.773755 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:58.775056 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:58.776161 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:58.776355 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:58.780335 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:58.780346 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:58.780378 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:58.780434 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:58.784332 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:58.785341 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:58.785349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:58.785356 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:58.785549 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:58.785983 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:58.785988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:58.785991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:58.786626 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:58.786929 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:58.787326 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 14:54:58.787684 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":346,"threads_started":1}
I20260619 14:54:58.792763 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:54:58.795923 22398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f87a839395943849e0adc4ed1f0a551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260619 14:54:58.796264 22398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:58.796370 22398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f87a839395943849e0adc4ed1f0a551, State: Initialized, Role: FOLLOWER
I20260619 14:54:58.796975 22398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f87a839395943849e0adc4ed1f0a551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260619 14:54:58.797271 22398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:58.797439 22398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:58.797613 22398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:58.800711 22398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f87a839395943849e0adc4ed1f0a551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260619 14:54:58.801275 22398 leader_election.cc:302] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f87a839395943849e0adc4ed1f0a551; no voters:
I20260619 14:54:58.801910 22398 leader_election.cc:288] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:58.802011 22408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:58.803726 22408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [term 1 LEADER]: Becoming Leader. State: Replica: 3f87a839395943849e0adc4ed1f0a551, State: Running, Role: LEADER
I20260619 14:54:58.804121 22398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:58.804646 22408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f87a839395943849e0adc4ed1f0a551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } }
I20260619 14:54:58.807893 22410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f87a839395943849e0adc4ed1f0a551. Latest consensus state: current_term: 1 leader_uuid: "3f87a839395943849e0adc4ed1f0a551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f87a839395943849e0adc4ed1f0a551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } } }
I20260619 14:54:58.808339 22410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:58.807832 22409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f87a839395943849e0adc4ed1f0a551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f87a839395943849e0adc4ed1f0a551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46307 } } }
I20260619 14:54:58.808570 22409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:58.816075 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:58.797031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:60607 (local address 127.0.0.1:46307)
0619 14:54:58.797227 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:58.797236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:58.797786 (+ 550us) server_negotiation.cc:407] Connection header received
0619 14:54:58.797987 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:58.797991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:58.798048 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:58.798122 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:58.799890 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:58.800581 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:58.802363 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:58.803073 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:58.803180 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:58.807244 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:58.807256 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:58.807262 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:58.807307 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:58.811336 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:58.812626 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:58.812635 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:58.812641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:58.812830 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:58.813567 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:58.813575 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:58.813580 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:58.814128 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:58.814458 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:58.815298 (+ 840us) server_negotiation.cc:299] Negotiation successful
0619 14:54:58.815557 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:54:58.819288 22415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:58.824026 22415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:58.832432 22415 catalog_manager.cc:1347] Generated new cluster ID: 0f58df55016749c1b79604f76f44a5d1
I20260619 14:54:58.832528 22415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:58.864277 22415 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:58.866153 22415 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:58.892475 22415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f87a839395943849e0adc4ed1f0a551: Generated new TSK 0
I20260619 14:54:58.893421 22415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:58.912626 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22331
I20260619 14:54:58.914144 22393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:54:59.013761 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 22331
W20260619 14:54:59.031008 22316 connection.cc:535] client connection to 127.0.0.1:46307 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46307): Connection reset by peer (error 104)
2026-06-19T14:54:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2034 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:59 dist-test-slave-p3ml krb5kdc[22430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:59 dist-test-slave-p3ml krb5kdc[22430](info): set up 2 sockets
Jun 19 14:54:59 dist-test-slave-p3ml krb5kdc[22430](info): commencing operation
krb5kdc: starting...
W20260619 14:55:00.115990 964 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:55:00 dist-test-slave-p3ml krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:00.354086 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:00.675064 22443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:00.675330 22443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:00.675520 22443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:00.682236 22443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:00.682328 22443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:00.682372 22443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:00.682459 22443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:00.682543 22443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:00.691949 22443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:00.693169 22443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:00.694425 22443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:00.706480 22451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:00.706470 22449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:00.707010 22448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:00.716151 22443 server_base.cc:1034] running on GCE node
I20260619 14:55:00.717252 22443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:00.719159 22443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:00.720407 22443 hybrid_clock.cc:648] HybridClock initialized: now 1781880900720310 us; error 71 us; skew 500 ppm
I20260619 14:55:00.720858 22443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:00 dist-test-slave-p3ml krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:00.724718 22443 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:00.726249 22443 webserver.cc:460] Webserver started at http://127.0.0.1:41651/ using document root <none> and password file <none>
I20260619 14:55:00.726953 22443 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:00.727098 22443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:00.727412 22443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:00.730216 22443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "3e2dfebbcb644da588f4b2ab1a4d6093"
format_stamp: "Formatted at 2026-06-19 14:55:00 on dist-test-slave-p3ml"
server_key: "f2ed28c31f8f0cd4ee8d058b2f741d25"
server_key_iv: "ae60dd3e2441fba245fb3ab22b913232"
server_key_version: "encryptionkey@0"
I20260619 14:55:00.730924 22443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "3e2dfebbcb644da588f4b2ab1a4d6093"
format_stamp: "Formatted at 2026-06-19 14:55:00 on dist-test-slave-p3ml"
server_key: "f2ed28c31f8f0cd4ee8d058b2f741d25"
server_key_iv: "ae60dd3e2441fba245fb3ab22b913232"
server_key_version: "encryptionkey@0"
I20260619 14:55:00.736001 22443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 14:55:00.739560 22458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:00.746394 22443 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260619 14:55:00.746575 22443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "3e2dfebbcb644da588f4b2ab1a4d6093"
format_stamp: "Formatted at 2026-06-19 14:55:00 on dist-test-slave-p3ml"
server_key: "f2ed28c31f8f0cd4ee8d058b2f741d25"
server_key_iv: "ae60dd3e2441fba245fb3ab22b913232"
server_key_version: "encryptionkey@0"
I20260619 14:55:00.746821 22443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:00.768455 22443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:00.773828 22443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:00.774135 22443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:00.793592 22443 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39457
I20260619 14:55:00.793676 22509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39457 every 8 connection(s)
I20260619 14:55:00.794921 22443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:00.796953 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 22443
I20260619 14:55:00.797160 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:55:00.797619 964 external_mini_cluster.cc:1428] Setting key d8c702e935a526fec4a72fa1055e370f
I20260619 14:55:00.799046 22510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:00.812144 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:00.799401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:32957 (local address 127.0.0.1:39457)
0619 14:55:00.800195 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:00.800209 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:00.800257 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:55:00.802224 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:00.802249 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:00.802729 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:00.803170 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:00.803772 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:00.805735 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:00.807085 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:00.808112 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:00.808288 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:00.809457 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:00.809472 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:00.809491 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:00.810199 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:00.810286 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:00.810320 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:00.810482 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:00.810699 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 14:55:00.810887 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":307,"threads_started":1}
I20260619 14:55:00.813359 22510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093: Bootstrap starting.
I20260619 14:55:00.816152 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:00.816946 22510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:00.818102 22510 log.cc:824] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:00.820886 22510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093: No bootstrap required, opened a new log
I20260619 14:55:00.827219 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:00.821185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:46543 (local address 127.0.0.1:39457)
0619 14:55:00.821384 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:00.821391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:00.821732 (+ 341us) server_negotiation.cc:407] Connection header received
0619 14:55:00.821953 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:00.821958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:00.822011 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:00.822118 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:00.822559 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:00.823358 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:00.824758 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:00.825222 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:00.825385 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:00.826519 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:00.826524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:00.826527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:00.826546 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:00.826582 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:00.826596 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:00.826712 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:00.826920 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 14:55:00.826973 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:55:00.829034 22510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2dfebbcb644da588f4b2ab1a4d6093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39457 } }
I20260619 14:55:00.829389 22510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:00.829501 22510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2dfebbcb644da588f4b2ab1a4d6093, State: Initialized, Role: FOLLOWER
I20260619 14:55:00.830034 22510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2dfebbcb644da588f4b2ab1a4d6093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39457 } }
I20260619 14:55:00.830255 22510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:00.830397 22510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:00.830559 22510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:00.833084 22510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2dfebbcb644da588f4b2ab1a4d6093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39457 } }
I20260619 14:55:00.833560 22510 leader_election.cc:302] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e2dfebbcb644da588f4b2ab1a4d6093; no voters:
I20260619 14:55:00.834121 22510 leader_election.cc:288] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:00.834208 22520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:00.834916 22520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [term 1 LEADER]: Becoming Leader. State: Replica: 3e2dfebbcb644da588f4b2ab1a4d6093, State: Running, Role: LEADER
I20260619 14:55:00.835583 22520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2dfebbcb644da588f4b2ab1a4d6093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39457 } }
I20260619 14:55:00.836403 22510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:00.838593 22522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e2dfebbcb644da588f4b2ab1a4d6093. Latest consensus state: current_term: 1 leader_uuid: "3e2dfebbcb644da588f4b2ab1a4d6093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2dfebbcb644da588f4b2ab1a4d6093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39457 } } }
I20260619 14:55:00.838562 22521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e2dfebbcb644da588f4b2ab1a4d6093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2dfebbcb644da588f4b2ab1a4d6093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39457 } } }
I20260619 14:55:00.839046 22522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:00.839085 22521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:00.839861 22528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:00.845870 22528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:00.856743 22528 catalog_manager.cc:1347] Generated new cluster ID: 4c41ef05fd83440a90a3ad34f39101f8
I20260619 14:55:00.856863 22528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:00.868058 22528 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:00.869793 22528 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:00.881297 22528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e2dfebbcb644da588f4b2ab1a4d6093: Generated new TSK 0
I20260619 14:55:00.882237 22528 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:00.889547 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22443
W20260619 14:55:00.890672 22505 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.56:32957
W20260619 14:55:00.891399 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22443: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:55:00.891582 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 22443
W20260619 14:55:00.908834 22428 connection.cc:535] client connection to 127.0.0.1:39457 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39457): Connection reset by peer (error 104)
2026-06-19T14:55:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:00 dist-test-slave-p3ml krb5kdc[22541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:00 dist-test-slave-p3ml krb5kdc[22541](info): set up 2 sockets
Jun 19 14:55:00 dist-test-slave-p3ml krb5kdc[22541](info): commencing operation
krb5kdc: starting...
W20260619 14:55:01.991758 964 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:55:02 dist-test-slave-p3ml krb5kdc[22541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:02.244966 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:02.571571 22554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:02.571908 22554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:02.572082 22554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:02.579196 22554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:02.579356 22554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:02.579495 22554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:02.579609 22554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:02.579699 22554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:02.589323 22554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45605
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:02.590611 22554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:02.591926 22554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:02.603592 22559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:02.603665 22560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:02.603613 22562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:02.604996 22554 server_base.cc:1034] running on GCE node
I20260619 14:55:02.605813 22554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:02.607151 22554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:02.608379 22554 hybrid_clock.cc:648] HybridClock initialized: now 1781880902608321 us; error 73 us; skew 500 ppm
I20260619 14:55:02.608724 22554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:02 dist-test-slave-p3ml krb5kdc[22541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:02.611779 22554 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:02.613294 22554 webserver.cc:460] Webserver started at http://127.0.0.1:45807/ using document root <none> and password file <none>
I20260619 14:55:02.613986 22554 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:02.614133 22554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:02.614439 22554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:02.617429 22554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "fee82632a25741a78094dc9f64a76cb4"
format_stamp: "Formatted at 2026-06-19 14:55:02 on dist-test-slave-p3ml"
server_key: "15accb9ec081c6ec8ac1d4db0c45651b"
server_key_iv: "c9c340755730c6c02d3ac94e5aa571f3"
server_key_version: "encryptionkey@0"
I20260619 14:55:02.618194 22554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "fee82632a25741a78094dc9f64a76cb4"
format_stamp: "Formatted at 2026-06-19 14:55:02 on dist-test-slave-p3ml"
server_key: "15accb9ec081c6ec8ac1d4db0c45651b"
server_key_iv: "c9c340755730c6c02d3ac94e5aa571f3"
server_key_version: "encryptionkey@0"
I20260619 14:55:02.623236 22554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 14:55:02.626708 22569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:02.627993 22554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:02.628178 22554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "fee82632a25741a78094dc9f64a76cb4"
format_stamp: "Formatted at 2026-06-19 14:55:02 on dist-test-slave-p3ml"
server_key: "15accb9ec081c6ec8ac1d4db0c45651b"
server_key_iv: "c9c340755730c6c02d3ac94e5aa571f3"
server_key_version: "encryptionkey@0"
I20260619 14:55:02.628374 22554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:02.633977 22554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:02.637764 22554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:02.638055 22554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:02.656143 22554 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45605
I20260619 14:55:02.656225 22620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45605 every 8 connection(s)
I20260619 14:55:02.657486 22554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:02.658720 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 22554
I20260619 14:55:02.658910 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:55:02.659492 964 external_mini_cluster.cc:1428] Setting key 3f86e1b4eaabecc6a0ebfef1266f4f31
I20260619 14:55:02.661978 22621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:02.666883 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:02.661201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:49783 (local address 127.0.0.1:45605)
0619 14:55:02.661955 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:02.661970 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:02.662012 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:55:02.663704 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:02.663725 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:02.664030 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:02.664336 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:02.665349 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:02.665354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:02.665366 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:02.665379 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:02.665446 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:02.665472 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:02.665636 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:02.665765 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 14:55:02.665977 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":333,"threads_started":1}
I20260619 14:55:02.670320 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:02.674355 22621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4: Bootstrap starting.
I20260619 14:55:02.677831 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:02.675496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:45857 (local address 127.0.0.1:45605)
0619 14:55:02.675670 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:02.675675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:02.675983 (+ 308us) server_negotiation.cc:407] Connection header received
0619 14:55:02.676148 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:02.676151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:02.676193 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:02.676269 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:02.677162 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:02.677165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:02.677168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:02.677181 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:02.677215 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:02.677229 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:02.677315 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:02.677476 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 14:55:02.677517 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:55:02.678368 22621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:02.679594 22621 log.cc:824] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:02.682503 22621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4: No bootstrap required, opened a new log
I20260619 14:55:02.689913 22621 raft_consensus.cc:348] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee82632a25741a78094dc9f64a76cb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45605 } }
I20260619 14:55:02.690227 22621 raft_consensus.cc:374] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:02.690296 22621 raft_consensus.cc:729] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee82632a25741a78094dc9f64a76cb4, State: Initialized, Role: FOLLOWER
I20260619 14:55:02.690783 22621 consensus_queue.cc:260] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee82632a25741a78094dc9f64a76cb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45605 } }
I20260619 14:55:02.690972 22621 raft_consensus.cc:388] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:02.691066 22621 raft_consensus.cc:482] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:02.691182 22621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:02.693243 22621 raft_consensus.cc:504] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee82632a25741a78094dc9f64a76cb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45605 } }
I20260619 14:55:02.693679 22621 leader_election.cc:302] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fee82632a25741a78094dc9f64a76cb4; no voters:
I20260619 14:55:02.694401 22621 leader_election.cc:288] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:02.694638 22631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:02.695919 22631 raft_consensus.cc:686] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [term 1 LEADER]: Becoming Leader. State: Replica: fee82632a25741a78094dc9f64a76cb4, State: Running, Role: LEADER
I20260619 14:55:02.696408 22631 consensus_queue.cc:237] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee82632a25741a78094dc9f64a76cb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45605 } }
I20260619 14:55:02.696542 22621 sys_catalog.cc:564] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:02.699119 22632 sys_catalog.cc:455] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fee82632a25741a78094dc9f64a76cb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee82632a25741a78094dc9f64a76cb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45605 } } }
I20260619 14:55:02.699481 22632 sys_catalog.cc:458] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:02.699887 22633 sys_catalog.cc:455] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fee82632a25741a78094dc9f64a76cb4. Latest consensus state: current_term: 1 leader_uuid: "fee82632a25741a78094dc9f64a76cb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fee82632a25741a78094dc9f64a76cb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45605 } } }
I20260619 14:55:02.700160 22633 sys_catalog.cc:458] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:02.704732 22638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:02.710201 22638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:02.718345 22638 catalog_manager.cc:1347] Generated new cluster ID: 30634305d83a4b34aaeafece0af41ae2
I20260619 14:55:02.718465 22638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:02.729656 22638 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:02.730077 22638 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:02.737432 22638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fee82632a25741a78094dc9f64a76cb4: Generated new TSK 0
I20260619 14:55:02.738416 22638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:02.756464 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22554
W20260619 14:55:02.757826 22616 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.56:49783
W20260619 14:55:02.758558 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22554: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:55:02.758723 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 22554
2026-06-19T14:55:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1867 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:02 dist-test-slave-p3ml krb5kdc[22652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:02 dist-test-slave-p3ml krb5kdc[22652](info): set up 2 sockets
Jun 19 14:55:02 dist-test-slave-p3ml krb5kdc[22652](info): commencing operation
krb5kdc: starting...
W20260619 14:55:03.867770 964 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:55:04 dist-test-slave-p3ml krb5kdc[22652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:55:04.123010 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:40867
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:42371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:40867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:04.454600 22665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:04.454885 22665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:04.455003 22665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:04.463783 22665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:04.463934 22665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:04.464032 22665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:04.464136 22665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:04.464210 22665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:04.479857 22665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:42371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:40867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:40867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:04.481760 22665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:04.483628 22665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:04.499370 22670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:04.499332 22671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:04.499987 22665 server_base.cc:1034] running on GCE node
W20260619 14:55:04.500056 22673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:04.500977 22665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:04.502408 22665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:04.503628 22665 hybrid_clock.cc:648] HybridClock initialized: now 1781880904503555 us; error 51 us; skew 500 ppm
I20260619 14:55:04.504025 22665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:04 dist-test-slave-p3ml krb5kdc[22652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:04.507484 22665 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:55:04.509370 22665 webserver.cc:460] Webserver started at http://127.0.241.62:37407/ using document root <none> and password file <none>
I20260619 14:55:04.510042 22665 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:04.510138 22665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:04.510377 22665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:04.513357 22665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "afba4c21febf461ca7aaa1ed91dfcd31"
format_stamp: "Formatted at 2026-06-19 14:55:04 on dist-test-slave-p3ml"
server_key: "353f49c46d80fd2abb72ab42efd06360"
server_key_iv: "ee53359f02a854215768c878c35250c7"
server_key_version: "encryptionkey@0"
I20260619 14:55:04.514071 22665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "afba4c21febf461ca7aaa1ed91dfcd31"
format_stamp: "Formatted at 2026-06-19 14:55:04 on dist-test-slave-p3ml"
server_key: "353f49c46d80fd2abb72ab42efd06360"
server_key_iv: "ee53359f02a854215768c878c35250c7"
server_key_version: "encryptionkey@0"
I20260619 14:55:04.518882 22665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:55:04.522210 22680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:04.523563 22665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 14:55:04.523694 22665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "afba4c21febf461ca7aaa1ed91dfcd31"
format_stamp: "Formatted at 2026-06-19 14:55:04 on dist-test-slave-p3ml"
server_key: "353f49c46d80fd2abb72ab42efd06360"
server_key_iv: "ee53359f02a854215768c878c35250c7"
server_key_version: "encryptionkey@0"
I20260619 14:55:04.523845 22665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:04.536711 22665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:04.540426 22665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:04.540692 22665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:04.559224 22665 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:40867
I20260619 14:55:04.559266 22731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:40867 every 8 connection(s)
I20260619 14:55:04.560662 22665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:04.564299 22732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:04.567756 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 22665
I20260619 14:55:04.567960 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:55:04.568495 964 external_mini_cluster.cc:1428] Setting key 1f1563ee47aad70091588168c5fa494a
I20260619 14:55:04.576876 22732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31: Bootstrap starting.
I20260619 14:55:04.580665 22732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:04.582007 22732 log.cc:824] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:55:04 dist-test-slave-p3ml krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:55:04.585155 22732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31: No bootstrap required, opened a new log
I20260619 14:55:04.591946 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:04.570246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51073 (local address 127.0.241.62:40867)
0619 14:55:04.571014 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:04.571024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:04.571064 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:55:04.572817 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:04.572847 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:04.573248 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:04.573564 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:04.575260 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:04.576566 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:04.578256 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:04.578907 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:04.579042 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:04.584420 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:04.584427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:04.584440 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:04.584473 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:04.587775 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:04.588639 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:04.588645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:04.588650 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:04.588837 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:04.589301 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:04.589304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:04.589307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:04.589805 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:04.590058 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:04.590552 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 14:55:04.590880 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":349,"threads_started":1}
I20260619 14:55:04.594348 22732 raft_consensus.cc:348] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afba4c21febf461ca7aaa1ed91dfcd31" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 40867 } }
I20260619 14:55:04.594764 22732 raft_consensus.cc:374] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:04.594888 22732 raft_consensus.cc:729] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afba4c21febf461ca7aaa1ed91dfcd31, State: Initialized, Role: FOLLOWER
I20260619 14:55:04.595592 22732 consensus_queue.cc:260] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afba4c21febf461ca7aaa1ed91dfcd31" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 40867 } }
I20260619 14:55:04.595826 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:04.595834 22732 raft_consensus.cc:388] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:04.595981 22732 raft_consensus.cc:482] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:04.596153 22732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:04.598264 22732 raft_consensus.cc:504] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afba4c21febf461ca7aaa1ed91dfcd31" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 40867 } }
I20260619 14:55:04.598826 22732 leader_election.cc:302] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afba4c21febf461ca7aaa1ed91dfcd31; no voters:
I20260619 14:55:04.599486 22732 leader_election.cc:288] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:04.599596 22741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:04.602075 22732 sys_catalog.cc:564] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:04.606243 22741 raft_consensus.cc:686] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [term 1 LEADER]: Becoming Leader. State: Replica: afba4c21febf461ca7aaa1ed91dfcd31, State: Running, Role: LEADER
I20260619 14:55:04.607021 22741 consensus_queue.cc:237] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afba4c21febf461ca7aaa1ed91dfcd31" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 40867 } }
I20260619 14:55:04.613224 22743 sys_catalog.cc:455] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afba4c21febf461ca7aaa1ed91dfcd31. Latest consensus state: current_term: 1 leader_uuid: "afba4c21febf461ca7aaa1ed91dfcd31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afba4c21febf461ca7aaa1ed91dfcd31" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 40867 } } }
I20260619 14:55:04.613606 22743 sys_catalog.cc:458] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:04.614203 22742 sys_catalog.cc:455] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afba4c21febf461ca7aaa1ed91dfcd31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afba4c21febf461ca7aaa1ed91dfcd31" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 40867 } } }
I20260619 14:55:04.614537 22742 sys_catalog.cc:458] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:04.618221 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:04.600818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56569 (local address 127.0.241.62:40867)
0619 14:55:04.601202 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:04.601209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:04.601228 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:55:04.601361 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:04.601366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:04.601436 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:04.601538 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:04.603121 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:04.604303 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:04.605785 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:04.606219 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:04.606376 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:04.609474 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:04.609486 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:04.609493 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:04.609540 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:04.613221 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:04.614220 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:04.614229 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:04.614234 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:04.614395 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:04.615119 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:04.615127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:04.615133 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:04.615711 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:04.616069 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:04.617445 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0619 14:55:04.617682 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
W20260619 14:55:04.625349 22758 catalog_manager.cc:1558] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:55:04.625453 22758 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:55:04.625587 22756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:04.627473 22756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:04.635412 22756 catalog_manager.cc:1347] Generated new cluster ID: 0f735eb223e748d892bb3cd322893a99
I20260619 14:55:04.635560 22756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:04.658802 22756 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:04.660190 22756 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:04.666401 22756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afba4c21febf461ca7aaa1ed91dfcd31: Generated new TSK 0
I20260619 14:55:04.667172 22756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:04.680676 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22665
I20260619 14:55:04.681787 22727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:55:04.773489 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 22665
W20260619 14:55:04.787678 22650 connection.cc:535] client connection to 127.0.241.62:40867 recv error: Network error: failed to read from TLS socket (remote: 127.0.241.62:40867): Connection reset by peer (error 104)
2026-06-19T14:55:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2012 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:04 dist-test-slave-p3ml krb5kdc[22764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:04 dist-test-slave-p3ml krb5kdc[22764](info): set up 2 sockets
Jun 19 14:55:04 dist-test-slave-p3ml krb5kdc[22764](info): commencing operation
krb5kdc: starting...
W20260619 14:55:05.948108 964 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:55:06 dist-test-slave-p3ml krb5kdc[22764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:55:06.186533 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:38409
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:35909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:38409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:06.512791 22777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:06.513046 22777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:06.513131 22777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:06.520100 22777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:06.520196 22777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:06.520256 22777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:06.520308 22777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:06.520350 22777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:06.529681 22777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:35909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:38409
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:38409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:06.530833 22777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:06.532085 22777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:06.543670 22783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:06.543708 22782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:06.543605 22785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:06.544646 22777 server_base.cc:1034] running on GCE node
I20260619 14:55:06.545372 22777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:06.546681 22777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:06.547916 22777 hybrid_clock.cc:648] HybridClock initialized: now 1781880906547864 us; error 44 us; skew 500 ppm
I20260619 14:55:06.548245 22777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:06 dist-test-slave-p3ml krb5kdc[22764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:06.551721 22777 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:55:06.553103 22777 webserver.cc:460] Webserver started at http://127.0.241.62:42105/ using document root <none> and password file <none>
I20260619 14:55:06.553768 22777 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:06.553922 22777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:06.554205 22777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:06.557109 22777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "c62ad903f816425bab9e08037b2b11e9"
format_stamp: "Formatted at 2026-06-19 14:55:06 on dist-test-slave-p3ml"
server_key: "f0547181bca709c337d2a5c1a508d9d3"
server_key_iv: "c3523b2923ad83ec196eb2d728287c6c"
server_key_version: "encryptionkey@0"
I20260619 14:55:06.557878 22777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "c62ad903f816425bab9e08037b2b11e9"
format_stamp: "Formatted at 2026-06-19 14:55:06 on dist-test-slave-p3ml"
server_key: "f0547181bca709c337d2a5c1a508d9d3"
server_key_iv: "c3523b2923ad83ec196eb2d728287c6c"
server_key_version: "encryptionkey@0"
I20260619 14:55:06.562956 22777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:55:06.566294 22792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:06.567682 22777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 14:55:06.567869 22777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "c62ad903f816425bab9e08037b2b11e9"
format_stamp: "Formatted at 2026-06-19 14:55:06 on dist-test-slave-p3ml"
server_key: "f0547181bca709c337d2a5c1a508d9d3"
server_key_iv: "c3523b2923ad83ec196eb2d728287c6c"
server_key_version: "encryptionkey@0"
I20260619 14:55:06.568055 22777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:06.584581 22777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:06.588876 22777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:06.589197 22777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:06.607293 22777 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:38409
I20260619 14:55:06.607326 22843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:38409 every 8 connection(s)
I20260619 14:55:06.608624 22777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:06.609730 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 22777
I20260619 14:55:06.609920 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:55:06.610471 964 external_mini_cluster.cc:1428] Setting key da7e5bab968d23e91df88feb8f22f3f9
I20260619 14:55:06.613202 22844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:06.625221 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:06.612207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53371 (local address 127.0.241.62:38409)
0619 14:55:06.612960 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:06.612974 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:06.613023 (+ 49us) server_negotiation.cc:407] Connection header received
0619 14:55:06.615303 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:06.615329 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:06.615829 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:06.616257 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:06.616771 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:06.618777 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:06.620188 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:06.621167 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:06.621358 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:06.622449 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:06.622468 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:06.622485 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:06.623226 (+ 741us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:06.623304 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:06.623338 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:06.623564 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:06.623804 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 14:55:06.623971 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":314,"threads_started":1}
I20260619 14:55:06.626737 22844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9: Bootstrap starting.
I20260619 14:55:06.628754 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:06.630033 22844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:06.631197 22844 log.cc:824] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:06.633922 22844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9: No bootstrap required, opened a new log
I20260619 14:55:06.640705 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:06.633995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50371 (local address 127.0.241.62:38409)
0619 14:55:06.634208 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:06.634217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:06.634620 (+ 403us) server_negotiation.cc:407] Connection header received
0619 14:55:06.634841 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:06.634845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:06.634899 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:06.634976 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:06.635573 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:06.636363 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:06.638182 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:06.638579 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:06.638667 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:06.639940 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:06.639944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:06.639947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:06.639970 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:06.640025 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:06.640042 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:06.640130 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:06.640333 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 14:55:06.640376 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:55:06.643095 22844 raft_consensus.cc:348] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62ad903f816425bab9e08037b2b11e9" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38409 } }
I20260619 14:55:06.643509 22844 raft_consensus.cc:374] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:06.643621 22844 raft_consensus.cc:729] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c62ad903f816425bab9e08037b2b11e9, State: Initialized, Role: FOLLOWER
I20260619 14:55:06.644207 22844 consensus_queue.cc:260] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62ad903f816425bab9e08037b2b11e9" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38409 } }
I20260619 14:55:06.644439 22844 raft_consensus.cc:388] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:06.644586 22844 raft_consensus.cc:482] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:06.644765 22844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:06.646849 22844 raft_consensus.cc:504] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62ad903f816425bab9e08037b2b11e9" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38409 } }
I20260619 14:55:06.647329 22844 leader_election.cc:302] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c62ad903f816425bab9e08037b2b11e9; no voters:
I20260619 14:55:06.647993 22844 leader_election.cc:288] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:06.648125 22854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:06.649035 22854 raft_consensus.cc:686] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [term 1 LEADER]: Becoming Leader. State: Replica: c62ad903f816425bab9e08037b2b11e9, State: Running, Role: LEADER
I20260619 14:55:06.649612 22854 consensus_queue.cc:237] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62ad903f816425bab9e08037b2b11e9" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38409 } }
I20260619 14:55:06.650185 22844 sys_catalog.cc:564] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:06.652406 22856 sys_catalog.cc:455] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c62ad903f816425bab9e08037b2b11e9. Latest consensus state: current_term: 1 leader_uuid: "c62ad903f816425bab9e08037b2b11e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62ad903f816425bab9e08037b2b11e9" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38409 } } }
I20260619 14:55:06.652822 22856 sys_catalog.cc:458] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:06.654055 22863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:06.653932 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c62ad903f816425bab9e08037b2b11e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62ad903f816425bab9e08037b2b11e9" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 38409 } } }
I20260619 14:55:06.654315 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:06.659600 22863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:06.667976 22863 catalog_manager.cc:1347] Generated new cluster ID: e45fd8d368d84b1bb0969c0423710c1c
I20260619 14:55:06.668094 22863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:06.679549 22863 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:06.680701 22863 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:06.696385 22863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c62ad903f816425bab9e08037b2b11e9: Generated new TSK 0
I20260619 14:55:06.697134 22863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:06.706938 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22777
W20260619 14:55:06.708738 22839 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53371
W20260619 14:55:06.709491 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22777: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:55:06.709645 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 22777
W20260619 14:55:06.726220 22762 connection.cc:535] client connection to 127.0.241.62:38409 recv error: Network error: failed to read from TLS socket (remote: 127.0.241.62:38409): Connection reset by peer (error 104)
2026-06-19T14:55:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:06 dist-test-slave-p3ml krb5kdc[22875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:06 dist-test-slave-p3ml krb5kdc[22875](info): set up 2 sockets
Jun 19 14:55:06 dist-test-slave-p3ml krb5kdc[22875](info): commencing operation
krb5kdc: starting...
W20260619 14:55:07.811826 964 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:55:07 dist-test-slave-p3ml krb5kdc[22875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:55:08.048350 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35271
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:35271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:08.378342 22888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:08.378671 22888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:08.378858 22888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:08.385908 22888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:08.386062 22888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:08.386174 22888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:08.386289 22888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:08.386384 22888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:08.395928 22888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:35271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:08.397205 22888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:08.398602 22888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:08.409900 22893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:08.410524 22894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:08.412140 22888 server_base.cc:1034] running on GCE node
W20260619 14:55:08.412319 22896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:08.413156 22888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:08.414553 22888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:08.415784 22888 hybrid_clock.cc:648] HybridClock initialized: now 1781880908415744 us; error 49 us; skew 500 ppm
I20260619 14:55:08.416136 22888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:08 dist-test-slave-p3ml krb5kdc[22875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:08.419119 22888 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:55:08.420738 22888 webserver.cc:460] Webserver started at http://127.0.241.62:38259/ using document root <none> and password file <none>
I20260619 14:55:08.421434 22888 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:08.421578 22888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:08.421909 22888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:08.424896 22888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "d867ba793ca6487dbbb1a482f8bdc99c"
format_stamp: "Formatted at 2026-06-19 14:55:08 on dist-test-slave-p3ml"
server_key: "25c4b05887402a001029f4e0dc2cd04c"
server_key_iv: "1e77c41aae5b56ff7004012016477149"
server_key_version: "encryptionkey@0"
I20260619 14:55:08.425675 22888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "d867ba793ca6487dbbb1a482f8bdc99c"
format_stamp: "Formatted at 2026-06-19 14:55:08 on dist-test-slave-p3ml"
server_key: "25c4b05887402a001029f4e0dc2cd04c"
server_key_iv: "1e77c41aae5b56ff7004012016477149"
server_key_version: "encryptionkey@0"
I20260619 14:55:08.430773 22888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 14:55:08.434207 22903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:08.435562 22888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:08.435729 22888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "d867ba793ca6487dbbb1a482f8bdc99c"
format_stamp: "Formatted at 2026-06-19 14:55:08 on dist-test-slave-p3ml"
server_key: "25c4b05887402a001029f4e0dc2cd04c"
server_key_iv: "1e77c41aae5b56ff7004012016477149"
server_key_version: "encryptionkey@0"
I20260619 14:55:08.435925 22888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:08.441660 22888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:08.445617 22888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:08.445899 22888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:08.464253 22888 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:35271
I20260619 14:55:08.464298 22954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:35271 every 8 connection(s)
I20260619 14:55:08.465631 22888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:08.469115 22955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:08.472707 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 22888
I20260619 14:55:08.472991 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:55:08.473443 964 external_mini_cluster.cc:1428] Setting key 0fee9a72ad6a002a3a03decaf606fa66
I20260619 14:55:08.481869 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:08.475221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34997 (local address 127.0.241.62:35271)
0619 14:55:08.476153 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:08.476167 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:08.476214 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:55:08.478141 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:08.478172 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:08.478628 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:08.479044 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:08.480005 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:08.480013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:08.480028 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:08.480044 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:08.480117 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:08.480146 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:08.480379 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:08.480509 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 14:55:08.480712 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":451,"threads_started":1}
I20260619 14:55:08.482708 22955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c: Bootstrap starting.
I20260619 14:55:08.485488 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:08.485934 22955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:08.487144 22955 log.cc:824] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:08.490414 22955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c: No bootstrap required, opened a new log
I20260619 14:55:08.493152 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:08.490688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48611 (local address 127.0.241.62:35271)
0619 14:55:08.490904 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:08.490913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:08.491254 (+ 341us) server_negotiation.cc:407] Connection header received
0619 14:55:08.491468 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:08.491476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:08.491530 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:08.491611 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:08.492487 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:08.492492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:08.492496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:08.492509 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:08.492543 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:08.492560 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:08.492654 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:08.492792 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 14:55:08.492835 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:55:08.498518 22955 raft_consensus.cc:348] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867ba793ca6487dbbb1a482f8bdc99c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35271 } }
I20260619 14:55:08.498881 22955 raft_consensus.cc:374] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:08.498993 22955 raft_consensus.cc:729] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d867ba793ca6487dbbb1a482f8bdc99c, State: Initialized, Role: FOLLOWER
I20260619 14:55:08.499614 22955 consensus_queue.cc:260] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867ba793ca6487dbbb1a482f8bdc99c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35271 } }
I20260619 14:55:08.499851 22955 raft_consensus.cc:388] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:08.499995 22955 raft_consensus.cc:482] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:08.500152 22955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:08.502166 22955 raft_consensus.cc:504] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867ba793ca6487dbbb1a482f8bdc99c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35271 } }
I20260619 14:55:08.502640 22955 leader_election.cc:302] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d867ba793ca6487dbbb1a482f8bdc99c; no voters:
I20260619 14:55:08.503162 22955 leader_election.cc:288] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:08.503288 22965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:08.504014 22965 raft_consensus.cc:686] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [term 1 LEADER]: Becoming Leader. State: Replica: d867ba793ca6487dbbb1a482f8bdc99c, State: Running, Role: LEADER
I20260619 14:55:08.504511 22965 consensus_queue.cc:237] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867ba793ca6487dbbb1a482f8bdc99c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35271 } }
I20260619 14:55:08.505173 22955 sys_catalog.cc:564] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:08.507233 22966 sys_catalog.cc:455] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d867ba793ca6487dbbb1a482f8bdc99c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867ba793ca6487dbbb1a482f8bdc99c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35271 } } }
I20260619 14:55:08.507587 22966 sys_catalog.cc:458] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:08.507723 22967 sys_catalog.cc:455] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d867ba793ca6487dbbb1a482f8bdc99c. Latest consensus state: current_term: 1 leader_uuid: "d867ba793ca6487dbbb1a482f8bdc99c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867ba793ca6487dbbb1a482f8bdc99c" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35271 } } }
I20260619 14:55:08.508030 22967 sys_catalog.cc:458] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:08.510093 22970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:08.515394 22970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:08.524124 22970 catalog_manager.cc:1347] Generated new cluster ID: 514f1eb4bc77483f81f6af6fdc9ed716
I20260619 14:55:08.524232 22970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:08.541410 22970 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:08.541819 22970 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:08.548090 22970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d867ba793ca6487dbbb1a482f8bdc99c: Generated new TSK 0
I20260619 14:55:08.548790 22970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:08.556507 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22888
W20260619 14:55:08.557624 22950 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34997
W20260619 14:55:08.558281 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22888: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:55:08.558434 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 22888
2026-06-19T14:55:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1848 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:08 dist-test-slave-p3ml krb5kdc[22986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:08 dist-test-slave-p3ml krb5kdc[22986](info): set up 2 sockets
Jun 19 14:55:08 dist-test-slave-p3ml krb5kdc[22986](info): commencing operation
krb5kdc: starting...
W20260619 14:55:09.671881 964 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:55:09 dist-test-slave-p3ml krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:55:09.916172 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45339
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:32901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:10.247136 22999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:10.247440 22999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:10.247633 22999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:10.254612 22999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:10.254715 22999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:10.254849 22999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:10.254962 22999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:10.255057 22999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:10.265120 22999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:32901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:45339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:10.266467 22999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:10.267856 22999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:10.280349 23005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:10.280494 23004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:10.280458 23007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:10.281098 22999 server_base.cc:1034] running on GCE node
I20260619 14:55:10.281895 22999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:10.283378 22999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:10.284638 22999 hybrid_clock.cc:648] HybridClock initialized: now 1781880910284581 us; error 47 us; skew 500 ppm
I20260619 14:55:10.284968 22999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:10 dist-test-slave-p3ml krb5kdc[22986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:10.292353 22999 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:55:10.294011 22999 webserver.cc:460] Webserver started at http://127.0.241.62:45165/ using document root <none> and password file <none>
I20260619 14:55:10.294739 22999 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:10.294905 22999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:10.295222 22999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:10.298324 22999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "7c760bb7b4c4447a91673ae2a5255133"
format_stamp: "Formatted at 2026-06-19 14:55:10 on dist-test-slave-p3ml"
server_key: "66f52aed593d08633e198b5f3a9fc2d8"
server_key_iv: "f2a04024a6c3498de8269f3e17010373"
server_key_version: "encryptionkey@0"
I20260619 14:55:10.299132 22999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "7c760bb7b4c4447a91673ae2a5255133"
format_stamp: "Formatted at 2026-06-19 14:55:10 on dist-test-slave-p3ml"
server_key: "66f52aed593d08633e198b5f3a9fc2d8"
server_key_iv: "f2a04024a6c3498de8269f3e17010373"
server_key_version: "encryptionkey@0"
I20260619 14:55:10.304325 22999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 14:55:10.307731 23014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:10.308933 22999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:55:10.309150 22999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "7c760bb7b4c4447a91673ae2a5255133"
format_stamp: "Formatted at 2026-06-19 14:55:10 on dist-test-slave-p3ml"
server_key: "66f52aed593d08633e198b5f3a9fc2d8"
server_key_iv: "f2a04024a6c3498de8269f3e17010373"
server_key_version: "encryptionkey@0"
I20260619 14:55:10.309352 22999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:10.324383 22999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:10.328011 22999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:10.328280 22999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:10.347939 22999 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:45339
I20260619 14:55:10.347975 23065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:45339 every 8 connection(s)
I20260619 14:55:10.349292 22999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:10.349546 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 22999
I20260619 14:55:10.349742 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:55:10.350381 964 external_mini_cluster.cc:1428] Setting key 4cdf00c7731722491433a17510b5e8f2
I20260619 14:55:10.353878 23066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 14:55:10 dist-test-slave-p3ml krb5kdc[22986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:55:10.366282 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133: Bootstrap starting.
I20260619 14:55:10.370334 23066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:10.371582 23066 log.cc:824] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:10.374271 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:10.352236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58931 (local address 127.0.241.62:45339)
0619 14:55:10.352965 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:10.352979 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:10.353026 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:55:10.355286 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:10.355313 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:10.355855 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:10.356247 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:10.357581 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:10.359614 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:10.360909 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:10.362032 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:10.362224 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:10.366191 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:10.366201 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:10.366217 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:10.366259 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:10.370057 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:10.370743 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:10.370756 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:10.370767 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:10.370960 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:10.371291 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:10.371298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:10.371301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:10.371957 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:10.372302 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:10.372676 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 14:55:10.373028 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":331,"threads_started":1}
I20260619 14:55:10.374874 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133: No bootstrap required, opened a new log
I20260619 14:55:10.377808 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:10.383294 23066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c760bb7b4c4447a91673ae2a5255133" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45339 } }
I20260619 14:55:10.383822 23066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:10.383924 23066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c760bb7b4c4447a91673ae2a5255133, State: Initialized, Role: FOLLOWER
I20260619 14:55:10.384625 23066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c760bb7b4c4447a91673ae2a5255133" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45339 } }
I20260619 14:55:10.384922 23066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:10.385087 23066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:10.385329 23066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:10.388334 23066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c760bb7b4c4447a91673ae2a5255133" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45339 } }
I20260619 14:55:10.388913 23066 leader_election.cc:302] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c760bb7b4c4447a91673ae2a5255133; no voters:
I20260619 14:55:10.389567 23066 leader_election.cc:288] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:10.389660 23076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:10.391109 23076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [term 1 LEADER]: Becoming Leader. State: Replica: 7c760bb7b4c4447a91673ae2a5255133, State: Running, Role: LEADER
I20260619 14:55:10.391841 23066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:10.391759 23076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c760bb7b4c4447a91673ae2a5255133" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45339 } }
I20260619 14:55:10.396843 23077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c760bb7b4c4447a91673ae2a5255133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c760bb7b4c4447a91673ae2a5255133" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45339 } } }
I20260619 14:55:10.397218 23077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:10.397697 23078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c760bb7b4c4447a91673ae2a5255133. Latest consensus state: current_term: 1 leader_uuid: "7c760bb7b4c4447a91673ae2a5255133" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c760bb7b4c4447a91673ae2a5255133" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45339 } } }
I20260619 14:55:10.398018 23078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:10.402453 23069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:10.382055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53917 (local address 127.0.241.62:45339)
0619 14:55:10.382256 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:10.382265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:10.382735 (+ 470us) server_negotiation.cc:407] Connection header received
0619 14:55:10.382918 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:10.382922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:10.382986 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:10.383061 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:10.385144 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:10.386049 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:10.387804 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:10.388227 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:10.388323 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:10.391425 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:10.391458 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:10.391463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:10.391498 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:10.396035 (+ 4537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:10.397128 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:10.397136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:10.397142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:10.397359 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:10.397951 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:10.397959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:10.397965 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:10.399925 (+ 1960us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:10.400274 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:10.401568 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0619 14:55:10.401867 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:55:10.407935 23086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:10.413100 23086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:10.421953 23086 catalog_manager.cc:1347] Generated new cluster ID: cd4c9c114d5d4dc7ad8321888db8ad6d
I20260619 14:55:10.422048 23086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:10.449935 23086 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:10.451913 23086 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:10.463085 23086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c760bb7b4c4447a91673ae2a5255133: Generated new TSK 0
I20260619 14:55:10.463923 23086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:10.477965 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 22999
I20260619 14:55:10.479601 23061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:55:10.573367 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 22999
2026-06-19T14:55:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2016 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:10 dist-test-slave-p3ml krb5kdc[23099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:10 dist-test-slave-p3ml krb5kdc[23099](info): set up 2 sockets
Jun 19 14:55:10 dist-test-slave-p3ml krb5kdc[23099](info): commencing operation
krb5kdc: starting...
W20260619 14:55:11.675779 964 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:55:11 dist-test-slave-p3ml krb5kdc[23099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:55:11.922775 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:36101
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:44885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:12.242434 23112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:12.242825 23112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:12.242996 23112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:12.250382 23112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:12.250545 23112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:12.250663 23112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:12.250784 23112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:12.250877 23112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:12.260795 23112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:44885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:36101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:12.262125 23112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:12.263545 23112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:12.276175 23118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:12.276265 23120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:12.276599 23117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:12.277328 23112 server_base.cc:1034] running on GCE node
I20260619 14:55:12.278147 23112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:12.279529 23112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:12.280735 23112 hybrid_clock.cc:648] HybridClock initialized: now 1781880912280698 us; error 32 us; skew 500 ppm
I20260619 14:55:12.281095 23112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:12 dist-test-slave-p3ml krb5kdc[23099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:12.284220 23112 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:55:12.285696 23112 webserver.cc:460] Webserver started at http://127.0.241.62:43523/ using document root <none> and password file <none>
I20260619 14:55:12.286387 23112 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:12.286535 23112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:12.286835 23112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:12.289831 23112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "03b5bee6bc974d42abc96f0c4165d1f1"
format_stamp: "Formatted at 2026-06-19 14:55:12 on dist-test-slave-p3ml"
server_key: "469df6db757a2127c78057b67624ef2a"
server_key_iv: "9c4fd3f0239fa5f81eb45db5768131b3"
server_key_version: "encryptionkey@0"
I20260619 14:55:12.290598 23112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "03b5bee6bc974d42abc96f0c4165d1f1"
format_stamp: "Formatted at 2026-06-19 14:55:12 on dist-test-slave-p3ml"
server_key: "469df6db757a2127c78057b67624ef2a"
server_key_iv: "9c4fd3f0239fa5f81eb45db5768131b3"
server_key_version: "encryptionkey@0"
I20260619 14:55:12.295614 23112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 14:55:12.298864 23127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:12.300168 23112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:12.300345 23112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "03b5bee6bc974d42abc96f0c4165d1f1"
format_stamp: "Formatted at 2026-06-19 14:55:12 on dist-test-slave-p3ml"
server_key: "469df6db757a2127c78057b67624ef2a"
server_key_iv: "9c4fd3f0239fa5f81eb45db5768131b3"
server_key_version: "encryptionkey@0"
I20260619 14:55:12.300541 23112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:12.316169 23112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:12.320158 23112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:12.320437 23112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:12.338399 23112 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:36101
I20260619 14:55:12.338429 23178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:36101 every 8 connection(s)
I20260619 14:55:12.339783 23112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:12.343324 23179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:12.346539 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 23112
I20260619 14:55:12.346753 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:55:12.347313 964 external_mini_cluster.cc:1428] Setting key 6cb7dcf15f500b0dedaa7d9c5c0ec500
I20260619 14:55:12.356423 23179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1: Bootstrap starting.
I20260619 14:55:12.359504 23179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:12.360586 23179 log.cc:824] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:12.361541 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:12.349551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59643 (local address 127.0.241.62:36101)
0619 14:55:12.351514 (+ 1963us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:12.351530 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:12.351582 (+ 52us) server_negotiation.cc:407] Connection header received
0619 14:55:12.353094 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:12.353114 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:12.353486 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:12.353877 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:12.354468 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:12.355828 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:12.357180 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:12.357844 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:12.358039 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:12.359232 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:12.359239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:12.359251 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:12.359874 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:12.359954 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:12.359981 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:12.360115 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:12.360342 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 14:55:12.360592 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1726,"thread_start_us":1504,"threads_started":1}
I20260619 14:55:12.363543 23179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1: No bootstrap required, opened a new log
I20260619 14:55:12.365129 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:12.371613 23179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b5bee6bc974d42abc96f0c4165d1f1" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36101 } }
I20260619 14:55:12.372035 23179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:12.372159 23179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b5bee6bc974d42abc96f0c4165d1f1, State: Initialized, Role: FOLLOWER
I20260619 14:55:12.372749 23179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b5bee6bc974d42abc96f0c4165d1f1" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36101 } }
I20260619 14:55:12.372978 23179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:12.373117 23179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:12.373282 23179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:12.375590 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:12.369287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58517 (local address 127.0.241.62:36101)
0619 14:55:12.369454 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:12.369460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:12.369837 (+ 377us) server_negotiation.cc:407] Connection header received
0619 14:55:12.369973 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:12.369978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:12.370054 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:12.370164 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:12.370610 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:12.371671 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:12.373044 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:12.373747 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:12.373899 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:12.374845 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:12.374864 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:12.374869 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:12.374887 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:12.374933 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:12.374951 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:12.375050 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:12.375139 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 14:55:12.375223 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:55:12.376368 23179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b5bee6bc974d42abc96f0c4165d1f1" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36101 } }
I20260619 14:55:12.376863 23179 leader_election.cc:302] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03b5bee6bc974d42abc96f0c4165d1f1; no voters:
I20260619 14:55:12.377396 23179 leader_election.cc:288] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:12.377535 23189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:12.378232 23189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [term 1 LEADER]: Becoming Leader. State: Replica: 03b5bee6bc974d42abc96f0c4165d1f1, State: Running, Role: LEADER
I20260619 14:55:12.378865 23189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b5bee6bc974d42abc96f0c4165d1f1" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36101 } }
I20260619 14:55:12.379585 23179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:12.381836 23190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03b5bee6bc974d42abc96f0c4165d1f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b5bee6bc974d42abc96f0c4165d1f1" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36101 } } }
I20260619 14:55:12.382157 23190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:12.383697 23195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:12.383744 23191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03b5bee6bc974d42abc96f0c4165d1f1. Latest consensus state: current_term: 1 leader_uuid: "03b5bee6bc974d42abc96f0c4165d1f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b5bee6bc974d42abc96f0c4165d1f1" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 36101 } } }
I20260619 14:55:12.384142 23191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:12.388986 23195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:12.397863 23195 catalog_manager.cc:1347] Generated new cluster ID: d8b2f266c6d84f7097050a57e8695c18
I20260619 14:55:12.397961 23195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:12.420639 23195 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:12.421911 23195 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:12.432797 23195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03b5bee6bc974d42abc96f0c4165d1f1: Generated new TSK 0
I20260619 14:55:12.433903 23195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:12.440594 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 23112
W20260619 14:55:12.441653 23174 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59643
W20260619 14:55:12.442282 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 23112: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:55:12.442545 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 23112
2026-06-19T14:55:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1867 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:12 dist-test-slave-p3ml krb5kdc[23210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:12 dist-test-slave-p3ml krb5kdc[23210](info): set up 2 sockets
Jun 19 14:55:12 dist-test-slave-p3ml krb5kdc[23210](info): commencing operation
krb5kdc: starting...
W20260619 14:55:13.547938 964 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:55:13 dist-test-slave-p3ml krb5kdc[23210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:55:13.792224 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:42689
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:42689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:14.119398 23223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:14.119767 23223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:14.119866 23223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:14.126931 23223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:14.127030 23223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:14.127076 23223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:14.127135 23223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:14.127180 23223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:14.136746 23223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:42689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:42689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:14.137872 23223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:14.139091 23223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:14.150605 23228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:14.150624 23231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:14.150667 23229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:14.151824 23223 server_base.cc:1034] running on GCE node
I20260619 14:55:14.152493 23223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:14.153743 23223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:14.154992 23223 hybrid_clock.cc:648] HybridClock initialized: now 1781880914154935 us; error 47 us; skew 500 ppm
I20260619 14:55:14.155333 23223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:14 dist-test-slave-p3ml krb5kdc[23210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:14.158419 23223 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:55:14.159940 23223 webserver.cc:460] Webserver started at http://127.0.241.62:35809/ using document root <none> and password file <none>
I20260619 14:55:14.160583 23223 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:14.160724 23223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:14.161010 23223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:14.164011 23223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "ab9dc2d444114efd9c1f5337009daa82"
format_stamp: "Formatted at 2026-06-19 14:55:14 on dist-test-slave-p3ml"
server_key: "af2a3ef062df8e79bec575d0f3c2f867"
server_key_iv: "ad66f889fc2891102f9334cd10544b5b"
server_key_version: "encryptionkey@0"
I20260619 14:55:14.164770 23223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "ab9dc2d444114efd9c1f5337009daa82"
format_stamp: "Formatted at 2026-06-19 14:55:14 on dist-test-slave-p3ml"
server_key: "af2a3ef062df8e79bec575d0f3c2f867"
server_key_iv: "ad66f889fc2891102f9334cd10544b5b"
server_key_version: "encryptionkey@0"
I20260619 14:55:14.169750 23223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:55:14.173038 23238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:14.174360 23223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:55:14.174538 23223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "ab9dc2d444114efd9c1f5337009daa82"
format_stamp: "Formatted at 2026-06-19 14:55:14 on dist-test-slave-p3ml"
server_key: "af2a3ef062df8e79bec575d0f3c2f867"
server_key_iv: "ad66f889fc2891102f9334cd10544b5b"
server_key_version: "encryptionkey@0"
I20260619 14:55:14.174736 23223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:14.180220 23223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:14.184427 23223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:14.184728 23223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:14.202917 23223 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:42689
I20260619 14:55:14.202962 23289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:42689 every 8 connection(s)
I20260619 14:55:14.204272 23223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:14.204696 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 23223
I20260619 14:55:14.204859 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:55:14.205437 964 external_mini_cluster.cc:1428] Setting key 850014da48f5a45394ef5ffad9e8d24d
I20260619 14:55:14.208454 23290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:14.212811 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:14.207129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33909 (local address 127.0.241.62:42689)
0619 14:55:14.207918 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:14.207928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:14.207968 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:55:14.209657 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:14.209677 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:14.209979 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:14.210278 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:14.211271 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:14.211277 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:14.211292 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:14.211306 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:14.211379 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:14.211398 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:14.211653 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:14.211746 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 14:55:14.211930 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":370,"threads_started":1}
I20260619 14:55:14.216254 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:14.220690 23290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82: Bootstrap starting.
I20260619 14:55:14.223793 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:14.221233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34909 (local address 127.0.241.62:42689)
0619 14:55:14.221401 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:14.221407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:14.221973 (+ 566us) server_negotiation.cc:407] Connection header received
0619 14:55:14.222106 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:14.222110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:14.222183 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:14.222313 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:14.223119 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:14.223124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:14.223128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:14.223146 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:14.223197 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:14.223216 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:14.223329 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:14.223406 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 14:55:14.223504 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:55:14.224437 23290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:14.225633 23290 log.cc:824] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:14.228220 23290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82: No bootstrap required, opened a new log
I20260619 14:55:14.235308 23290 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9dc2d444114efd9c1f5337009daa82" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 42689 } }
I20260619 14:55:14.235702 23290 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:14.235774 23290 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9dc2d444114efd9c1f5337009daa82, State: Initialized, Role: FOLLOWER
I20260619 14:55:14.236225 23290 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9dc2d444114efd9c1f5337009daa82" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 42689 } }
I20260619 14:55:14.236405 23290 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:14.236500 23290 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:14.236644 23290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:14.238624 23290 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9dc2d444114efd9c1f5337009daa82" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 42689 } }
I20260619 14:55:14.239027 23290 leader_election.cc:302] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab9dc2d444114efd9c1f5337009daa82; no voters:
I20260619 14:55:14.239643 23290 leader_election.cc:288] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:14.239710 23300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:14.240756 23300 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [term 1 LEADER]: Becoming Leader. State: Replica: ab9dc2d444114efd9c1f5337009daa82, State: Running, Role: LEADER
I20260619 14:55:14.241285 23300 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9dc2d444114efd9c1f5337009daa82" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 42689 } }
I20260619 14:55:14.241840 23290 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:14.244298 23301 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab9dc2d444114efd9c1f5337009daa82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9dc2d444114efd9c1f5337009daa82" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 42689 } } }
I20260619 14:55:14.244270 23302 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9dc2d444114efd9c1f5337009daa82. Latest consensus state: current_term: 1 leader_uuid: "ab9dc2d444114efd9c1f5337009daa82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9dc2d444114efd9c1f5337009daa82" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 42689 } } }
I20260619 14:55:14.245282 23301 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:14.245303 23302 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:14.246351 23307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:14.251425 23307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:14.260075 23307 catalog_manager.cc:1347] Generated new cluster ID: 49f2cb6d6632406db5cac77cd2eb49c6
I20260619 14:55:14.260197 23307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:14.279223 23307 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:14.279680 23307 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:14.285584 23307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab9dc2d444114efd9c1f5337009daa82: Generated new TSK 0
I20260619 14:55:14.286294 23307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:14.291656 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 23223
W20260619 14:55:14.292691 23285 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33909
W20260619 14:55:14.293296 964 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 23223: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 14:55:14.293443 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 23223
2026-06-19T14:55:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1847 ms)
[----------] 18 tests from AuthTokenIssuingTest (34572 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:14 dist-test-slave-p3ml krb5kdc[23321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:14 dist-test-slave-p3ml krb5kdc[23321](info): set up 2 sockets
Jun 19 14:55:14 dist-test-slave-p3ml krb5kdc[23321](info): commencing operation
krb5kdc: starting...
W20260619 14:55:15.403986 964 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:55:15 dist-test-slave-p3ml krb5kdc[23321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:55:15.644860 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45685
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:45685,127.0.241.61:44625,127.0.241.60:37053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:15.960184 23334 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:55:15.960529 23334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:15.960651 23334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:15.960788 23334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:15.967799 23334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:15.967908 23334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:15.968016 23334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:15.968111 23334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:15.968199 23334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:15.978031 23334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:45685,127.0.241.61:44625,127.0.241.60:37053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:45685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:15.979295 23334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:15.980554 23334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:15.992072 23339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:15.992166 23342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:15.992136 23340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:15.992865 23334 server_base.cc:1034] running on GCE node
I20260619 14:55:15.993580 23334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:15.994858 23334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:15.996101 23334 hybrid_clock.cc:648] HybridClock initialized: now 1781880915996041 us; error 55 us; skew 500 ppm
I20260619 14:55:15.996492 23334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:15 dist-test-slave-p3ml krb5kdc[23321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:16.004142 23334 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:55:16.005472 23334 webserver.cc:460] Webserver started at http://127.0.241.62:44117/ using document root <none> and password file <none>
I20260619 14:55:16.006085 23334 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:16.006181 23334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:16.006443 23334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:16.009372 23334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "9b11fb47e4f449dfbed4e50c9abd56e3"
format_stamp: "Formatted at 2026-06-19 14:55:15 on dist-test-slave-p3ml"
server_key: "f6eec3eff129725736c6ae4bbe94a3fb"
server_key_iv: "7cfd7dfce17bcac7d35cc90a8af126a3"
server_key_version: "encryptionkey@0"
I20260619 14:55:16.010076 23334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "9b11fb47e4f449dfbed4e50c9abd56e3"
format_stamp: "Formatted at 2026-06-19 14:55:15 on dist-test-slave-p3ml"
server_key: "f6eec3eff129725736c6ae4bbe94a3fb"
server_key_iv: "7cfd7dfce17bcac7d35cc90a8af126a3"
server_key_version: "encryptionkey@0"
I20260619 14:55:16.014994 23334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 14:55:16.018321 23349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:16.019676 23334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:16.019850 23334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "9b11fb47e4f449dfbed4e50c9abd56e3"
format_stamp: "Formatted at 2026-06-19 14:55:15 on dist-test-slave-p3ml"
server_key: "f6eec3eff129725736c6ae4bbe94a3fb"
server_key_iv: "7cfd7dfce17bcac7d35cc90a8af126a3"
server_key_version: "encryptionkey@0"
I20260619 14:55:16.020056 23334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:16.036448 23334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:16.040719 23334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:16.040994 23334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:16.059477 23334 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:45685
I20260619 14:55:16.059505 23400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:45685 every 8 connection(s)
I20260619 14:55:16.060814 23334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:16.064302 23401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:16.067940 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 23334
I20260619 14:55:16.068151 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:55:16.068971 964 external_mini_cluster.cc:1428] Setting key dcc4e9c5db03587d1cec846194be89d1
I20260619 14:55:16.072654 23401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:16 dist-test-slave-p3ml krb5kdc[23321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.241.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.61@KRBTEST.COM" created.
I20260619 14:55:16.097787 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.077638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.0.241.62:45685)
0619 14:55:16.078377 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:16.078391 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:16.078436 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:55:16.080570 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:16.080576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:16.080788 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:16.080934 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:16.082602 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.083826 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.086112 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.086651 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.086779 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:16.090805 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:16.090813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:16.090823 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:16.090849 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:16.093579 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:16.094213 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:16.094227 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:16.094230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:16.094372 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:16.094772 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:16.094777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:16.094787 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:16.095274 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:16.095503 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:16.096002 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 14:55:16.096331 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":346,"threads_started":1}
I20260619 14:55:16.097823 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.076329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45685 (local address 127.0.0.1:60118)
0619 14:55:16.077190 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.077240 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:16.080201 (+ 2961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:16.081020 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:16.081029 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:16.081836 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:16.082380 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:16.082407 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.083987 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.083995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:16.085883 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:16.085901 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.086889 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.086898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:16.087133 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:16.087850 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:16.087882 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:16.090579 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:16.093700 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:16.093712 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:16.093723 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:16.094090 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:16.094485 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:16.094489 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:16.094493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:16.094642 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:16.095414 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:16.095421 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:16.095670 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 14:55:16.095909 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 14:55:16.096477 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":133,"thread_start_us":366,"threads_started":1}
I20260619 14:55:16.100886 23401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:55:16.101897 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.101532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39292)
0619 14:55:16.101694 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.101801 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 14:55:16.102190 23353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
I20260619 14:55:16.103729 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.103356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39298)
0619 14:55:16.103534 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.103595 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 14:55:16.103911 23401 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:44625: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.61.keytab.
Entry for principal kudu/127.0.241.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.61.keytab.
I20260619 14:55:16.126518 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.61:44625
--webserver_interface=127.0.241.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.61.keytab
--principal=kudu/127.0.241.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:45685,127.0.241.61:44625,127.0.241.60:37053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:55:16.153230 23401 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } attempt: 1
I20260619 14:55:16.154433 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.153944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39314)
0619 14:55:16.154152 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.154280 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 14:55:16.155514 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.155075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39324)
0619 14:55:16.155219 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.155298 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 14:55:16.155732 23401 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:44625: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
I20260619 14:55:16.223990 23401 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } attempt: 2
I20260619 14:55:16.225214 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.224703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39334)
0619 14:55:16.224920 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.225045 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 14:55:16.226298 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.225880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39350)
0619 14:55:16.226051 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.226148 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 14:55:16.226534 23401 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:44625: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
I20260619 14:55:16.317895 23401 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } attempt: 3
I20260619 14:55:16.319093 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.318669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39354)
0619 14:55:16.318861 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.318952 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 14:55:16.320227 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.319880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39364)
0619 14:55:16.320016 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.320084 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 14:55:16.320518 23401 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:44625: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
W20260619 14:55:16.451622 23408 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:55:16.451996 23408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:16.452124 23408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:16.452265 23408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:16.459370 23408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:16.459534 23408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:16.459659 23408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:16.459755 23408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:16.459844 23408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:16.463747 23401 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } attempt: 4
I20260619 14:55:16.464985 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.464445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39376)
0619 14:55:16.464766 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.464858 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260619 14:55:16.465941 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.465598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39390)
0619 14:55:16.465753 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.465824 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 14:55:16.466101 23401 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:44625: Network error: Client connection negotiation failed: client connection to 127.0.241.61:44625: connect: Connection refused (error 111)
I20260619 14:55:16.470149 23408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:45685,127.0.241.61:44625,127.0.241.60:37053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.61.keytab
--principal=kudu/127.0.241.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.61:44625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.241.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:16.471503 23408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:16.472779 23408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:16.484726 23413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:16.484956 23414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:16.485121 23416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:16.485625 23408 server_base.cc:1034] running on GCE node
I20260619 14:55:16.486351 23408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:16.487681 23408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:16.488889 23408 hybrid_clock.cc:648] HybridClock initialized: now 1781880916488813 us; error 51 us; skew 500 ppm
I20260619 14:55:16.489202 23408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:16 dist-test-slave-p3ml krb5kdc[23321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:16.492249 23408 init.cc:377] Logged in from keytab as kudu/127.0.241.61@KRBTEST.COM (short username kudu)
I20260619 14:55:16.493503 23408 webserver.cc:460] Webserver started at http://127.0.241.61:39515/ using document root <none> and password file <none>
I20260619 14:55:16.494127 23408 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:16.494210 23408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:16.494454 23408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:16.497253 23408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/data/instance:
uuid: "a49c15caeed94e7694e21033d42ae208"
format_stamp: "Formatted at 2026-06-19 14:55:16 on dist-test-slave-p3ml"
server_key: "a83c1bbcbeb2ce495d69a51f1e0f1da2"
server_key_iv: "23dea8d049b698d25442e100d8b636b8"
server_key_version: "encryptionkey@0"
I20260619 14:55:16.497927 23408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/wal/instance:
uuid: "a49c15caeed94e7694e21033d42ae208"
format_stamp: "Formatted at 2026-06-19 14:55:16 on dist-test-slave-p3ml"
server_key: "a83c1bbcbeb2ce495d69a51f1e0f1da2"
server_key_iv: "23dea8d049b698d25442e100d8b636b8"
server_key_version: "encryptionkey@0"
I20260619 14:55:16.502574 23408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:55:16.505688 23423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:16.506804 23408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:16.506964 23408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/wal
uuid: "a49c15caeed94e7694e21033d42ae208"
format_stamp: "Formatted at 2026-06-19 14:55:16 on dist-test-slave-p3ml"
server_key: "a83c1bbcbeb2ce495d69a51f1e0f1da2"
server_key_iv: "23dea8d049b698d25442e100d8b636b8"
server_key_version: "encryptionkey@0"
I20260619 14:55:16.507162 23408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:16.537001 23408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:16.540998 23408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:16.541265 23408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:16.559187 23408 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.61:44625
I20260619 14:55:16.559227 23474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.61:44625 every 8 connection(s)
I20260619 14:55:16.560495 23408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
I20260619 14:55:16.564203 23475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:16.569373 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 23408
I20260619 14:55:16.569564 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-1/wal/instance
I20260619 14:55:16.570001 964 external_mini_cluster.cc:1428] Setting key 821631969498e46377438f3534253788
I20260619 14:55:16.570755 23475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:16 dist-test-slave-p3ml krb5kdc[23321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.61@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:55:16.591781 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.573358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.0.241.62:45685)
0619 14:55:16.573597 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:16.573605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:16.574736 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 14:55:16.576642 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:16.576652 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:16.576746 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:16.576866 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:16.578289 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.578964 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.581418 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.581863 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.581964 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:16.586041 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:16.586046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:16.586049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:16.586082 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:16.588826 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:16.589544 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:16.589549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:16.589551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:16.589675 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:16.590078 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:16.590085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:16.590093 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:16.590362 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:16.590516 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:16.591188 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 14:55:16.591425 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:55:16.592718 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.573733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45685 (local address 127.0.0.1:60120)
0619 14:55:16.574563 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.574606 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:16.576370 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:16.577045 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:16.577052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:16.577552 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:16.578095 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:16.578111 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.579132 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.579150 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:16.581267 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:16.581289 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.582096 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.582105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:16.582433 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:16.583157 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:16.583189 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:16.585819 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:16.588962 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:16.588971 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:16.588987 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:16.589412 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:16.589785 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:16.589791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:16.589795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:16.589960 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:16.590492 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:16.590499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:16.590837 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 14:55:16.591098 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 14:55:16.591614 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":391,"threads_started":1}
I20260619 14:55:16.594470 23475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.241.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.60@KRBTEST.COM" created.
Jun 19 14:55:16 dist-test-slave-p3ml krb5kdc[23321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.61@KRBTEST.COM for kudu/127.0.241.61@KRBTEST.COM
I20260619 14:55:16.609093 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.595345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.0.241.61:44625)
0619 14:55:16.595977 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:16.595982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:16.596004 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:55:16.596044 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:16.596049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:16.596213 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:16.596373 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:16.597110 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.598219 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.599398 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.599972 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.600107 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:16.603023 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:16.603038 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:16.603057 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:16.603099 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:16.606237 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:16.607059 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:16.607065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:16.607067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:16.607171 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:16.607601 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:16.607604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:16.607606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:16.607901 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:16.608056 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:16.608503 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 14:55:16.608785 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":435,"thread_start_us":293,"threads_started":1}
I20260619 14:55:16.609201 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.595314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39406)
0619 14:55:16.595549 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.595572 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:16.595827 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:16.596357 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:16.596362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:16.596776 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:16.596912 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:16.596919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.598355 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.598362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:16.599262 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:16.599274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.600125 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.600130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:16.600306 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:16.600856 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:16.600873 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:16.602796 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:16.606373 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:16.606381 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:16.606388 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:16.606937 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:16.607278 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:16.607286 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:16.607292 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:16.607493 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:16.608037 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:16.608044 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:16.608218 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 14:55:16.608370 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 14:55:16.608676 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 14:55:16.611588 23475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:55:16.612751 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.612334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53374)
0619 14:55:16.612502 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.612625 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 14:55:16.613060 23425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
I20260619 14:55:16.614621 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.614269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53384)
0619 14:55:16.614431 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.614504 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 14:55:16.614880 23475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37053: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.60.keytab.
Entry for principal kudu/127.0.241.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.60.keytab.
I20260619 14:55:16.628181 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.60:37053
--webserver_interface=127.0.241.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.60.keytab
--principal=kudu/127.0.241.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:46613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:45685,127.0.241.61:44625,127.0.241.60:37053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:55:16.664150 23475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } attempt: 1
I20260619 14:55:16.665323 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.664888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53396)
0619 14:55:16.665078 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.665177 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 14:55:16.666272 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.665953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53398)
0619 14:55:16.666107 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.666178 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 14:55:16.666455 23475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37053: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
I20260619 14:55:16.734755 23475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } attempt: 2
I20260619 14:55:16.735945 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.735508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53412)
0619 14:55:16.735713 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.735799 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 14:55:16.736882 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.736571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53428)
0619 14:55:16.736709 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.736775 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 14:55:16.737092 23475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37053: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
I20260619 14:55:16.765357 23401 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } attempt: 5
Jun 19 14:55:16 dist-test-slave-p3ml krb5kdc[23321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for kudu/127.0.241.61@KRBTEST.COM
I20260619 14:55:16.779747 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.766179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39420)
0619 14:55:16.766413 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.766433 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:16.766599 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:16.766985 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:16.766990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:16.767411 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:16.767660 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:16.767671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.768673 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.768683 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:16.769500 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:16.769511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.770266 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.770272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:16.770469 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:16.771204 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:16.771222 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:16.773605 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:16.777522 (+ 3917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:16.777526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:16.777529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:16.777896 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:16.778297 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:16.778300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:16.778302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:16.778419 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:16.779001 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:16.779004 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:16.779144 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 14:55:16.779251 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:55:16.779506 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 14:55:16.780184 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.766219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.0.241.61:44625)
0619 14:55:16.766419 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:16.766428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:16.766524 (+ 96us) server_negotiation.cc:407] Connection header received
0619 14:55:16.766749 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:16.766754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:16.766833 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:16.766946 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:16.767816 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.768519 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.769653 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:16.770109 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:16.770246 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:16.773794 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:16.773798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:16.773802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:16.773832 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:16.777382 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:16.778017 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:16.778022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:16.778026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:16.778188 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:16.778523 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:16.778527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:16.778531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:16.778895 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:16.779046 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:16.779428 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 14:55:16.779745 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 14:55:16.781266 23401 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:55:16.782303 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.781930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53440)
0619 14:55:16.782102 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.782176 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 14:55:16.783222 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.782906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53450)
0619 14:55:16.783058 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.783138 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 14:55:16.783411 23401 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37053: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
I20260619 14:55:16.828295 23475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } attempt: 3
I20260619 14:55:16.829402 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.828913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53466)
0619 14:55:16.829159 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.829245 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 14:55:16.830494 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.830087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53468)
0619 14:55:16.830308 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.830392 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260619 14:55:16.830684 23475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37053: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
I20260619 14:55:16.834643 23401 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } attempt: 1
I20260619 14:55:16.835588 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.835138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53472)
0619 14:55:16.835303 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.835392 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:55:16.836525 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.836214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53482)
0619 14:55:16.836360 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.836429 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:55:16.836719 23401 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37053: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
I20260619 14:55:16.904963 23401 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } attempt: 2
I20260619 14:55:16.906102 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.905613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53492)
0619 14:55:16.905849 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.905942 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 14:55:16.907102 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.906791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53500)
0619 14:55:16.906949 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.907010 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 14:55:16.907306 23401 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37053: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
W20260619 14:55:16.956391 23482 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:55:16.956768 23482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:16.956893 23482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:16.957022 23482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:16.964136 23482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:16.964257 23482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:16.964365 23482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:16.964450 23482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:16.964555 23482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:16.973893 23482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:46613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:45685,127.0.241.61:44625,127.0.241.60:37053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/krb5kdc/kudu_127.0.241.60.keytab
--principal=kudu/127.0.241.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.60:37053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.241.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:16.973882 23475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } attempt: 4
I20260619 14:55:16.974913 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.974445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53512)
0619 14:55:16.974678 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.974761 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 14:55:16.975237 23482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:16.975993 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.975673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53522)
0619 14:55:16.975820 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.975896 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 14:55:16.976238 23475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37053: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
I20260619 14:55:16.976675 23482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:16.995210 23490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:16.995250 23487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:16.995159 23488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:16.996618 23482 server_base.cc:1034] running on GCE node
I20260619 14:55:16.997308 23482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:16.998677 23482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:16.999917 23482 hybrid_clock.cc:648] HybridClock initialized: now 1781880916999852 us; error 62 us; skew 500 ppm
I20260619 14:55:17.000262 23482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:16 dist-test-slave-p3ml krb5kdc[23321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:17.003228 23482 init.cc:377] Logged in from keytab as kudu/127.0.241.60@KRBTEST.COM (short username kudu)
I20260619 14:55:17.004680 23482 webserver.cc:460] Webserver started at http://127.0.241.60:36505/ using document root <none> and password file <none>
I20260619 14:55:17.005313 23482 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:17.005398 23482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:17.005663 23482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:17.008541 23482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/data/instance:
uuid: "d14f69d073f644efa72a52413a101ef2"
format_stamp: "Formatted at 2026-06-19 14:55:17 on dist-test-slave-p3ml"
server_key: "53b12d0869dc00ffd7d4e0fe3dd43cb6"
server_key_iv: "49231373518fdec2719f4be57f59c026"
server_key_version: "encryptionkey@0"
I20260619 14:55:17.009239 23482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/wal/instance:
uuid: "d14f69d073f644efa72a52413a101ef2"
format_stamp: "Formatted at 2026-06-19 14:55:17 on dist-test-slave-p3ml"
server_key: "53b12d0869dc00ffd7d4e0fe3dd43cb6"
server_key_iv: "49231373518fdec2719f4be57f59c026"
server_key_version: "encryptionkey@0"
I20260619 14:55:17.013514 23401 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } attempt: 3
I20260619 14:55:17.014374 23482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:55:17.014600 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.014104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53536)
0619 14:55:17.014322 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:17.014402 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 14:55:17.015671 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.015258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53544)
0619 14:55:17.015492 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:17.015556 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260619 14:55:17.015856 23401 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:37053: Network error: Client connection negotiation failed: client connection to 127.0.241.60:37053: connect: Connection refused (error 111)
I20260619 14:55:17.018283 23497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:17.019677 23482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 14:55:17.019860 23482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/wal
uuid: "d14f69d073f644efa72a52413a101ef2"
format_stamp: "Formatted at 2026-06-19 14:55:17 on dist-test-slave-p3ml"
server_key: "53b12d0869dc00ffd7d4e0fe3dd43cb6"
server_key_iv: "49231373518fdec2719f4be57f59c026"
server_key_version: "encryptionkey@0"
I20260619 14:55:17.020085 23482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:17.061084 23482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:17.065438 23482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:17.065727 23482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:17.085196 23482 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.60:37053
I20260619 14:55:17.085204 23548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.60:37053 every 8 connection(s)
I20260619 14:55:17.086495 23482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
I20260619 14:55:17.090086 23549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:17.092039 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 23482
I20260619 14:55:17.092234 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880565277815-964-0/minicluster-data/master-2/wal/instance
I20260619 14:55:17.092715 964 external_mini_cluster.cc:1428] Setting key 799b072243f62ad5fdfecad417fe169c
I20260619 14:55:17.097436 23549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:17 dist-test-slave-p3ml krb5kdc[23321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.60@KRBTEST.COM
Jun 19 14:55:17 dist-test-slave-p3ml krb5kdc[23321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.60@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:55:17.114504 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.100307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.0.241.62:45685)
0619 14:55:17.101614 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.101619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.101639 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:55:17.101704 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.101712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.101795 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.101919 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.103009 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.103787 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.105070 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.105493 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.105627 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.108873 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:17.108878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:17.108881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:17.108911 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:17.111668 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.112445 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.112449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.112451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.112578 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.112952 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.112956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.112957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.113185 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:17.113369 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:17.113832 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 14:55:17.114057 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1190,"thread_start_us":1075,"threads_started":1}
I20260619 14:55:17.115127 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.094606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.0.241.60:37053)
0619 14:55:17.095402 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.095412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.095506 (+ 94us) server_negotiation.cc:407] Connection header received
0619 14:55:17.097117 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.097138 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.097503 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.097829 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.099593 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.101625 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.102956 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.103621 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.103798 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.107945 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:17.107953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:17.107962 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:17.107989 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:17.110982 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.111703 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.111710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.111719 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.111942 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.112334 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.112340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.112343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.112955 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:17.113224 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:17.113642 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 14:55:17.114053 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":350,"threads_started":1}
I20260619 14:55:17.115166 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.100286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:45685 (local address 127.0.0.1:60124)
0619 14:55:17.101083 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:17.101124 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:17.101290 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:17.101981 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:17.101987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:17.102617 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:17.102857 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:17.102865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.103943 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.103948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:17.104915 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:17.104927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.105728 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.105736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:17.105880 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.106450 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:17.106477 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:17.108678 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:17.111816 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:17.111826 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:17.111834 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:17.112334 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:17.112690 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:17.112694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:17.112697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:17.112849 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:17.113320 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:17.113327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:17.113535 (+ 208us) client_negotiation.cc:769] Sending connection context
0619 14:55:17.113739 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 14:55:17.114244 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":149,"thread_start_us":356,"threads_started":1}
I20260619 14:55:17.117460 23549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:17 dist-test-slave-p3ml krb5kdc[23321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.60@KRBTEST.COM for kudu/127.0.241.61@KRBTEST.COM
Jun 19 14:55:17 dist-test-slave-p3ml krb5kdc[23321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:55:17.132304 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.118172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:44625 (local address 127.0.0.1:39434)
0619 14:55:17.118424 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:17.118440 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:17.118558 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:17.118908 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:17.118917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:17.119226 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:17.119346 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:17.119353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.120585 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.120590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:17.121587 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:17.121598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.122480 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.122487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:17.122698 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.123313 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:17.123353 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:17.126145 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:17.129854 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:17.129858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:17.129860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:17.130211 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:17.130584 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:17.130587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:17.130589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:17.130721 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:17.131343 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:17.131347 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:17.131527 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 14:55:17.131690 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:55:17.131946 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260619 14:55:17.132340 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.118215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.0.241.61:44625)
0619 14:55:17.118394 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.118400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.118516 (+ 116us) server_negotiation.cc:407] Connection header received
0619 14:55:17.118680 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.118683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.118770 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.118866 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.119519 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.120438 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.121721 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.122346 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.122465 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.126320 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:17.126324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:17.126328 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:17.126381 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:17.129734 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.130325 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.130329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.130332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.130479 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.130847 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.130868 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.130872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.131218 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:17.131377 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:17.131777 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 14:55:17.131972 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:55:17.133535 23549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:55:17.133709 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.118142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.0.241.62:45685)
0619 14:55:17.118293 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.118298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.118402 (+ 104us) server_negotiation.cc:407] Connection header received
0619 14:55:17.118573 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.118578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.118668 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.118747 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.120124 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.120945 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.122461 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.123117 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.123231 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.127887 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:17.127897 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:17.127902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:17.127945 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:17.131054 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.131700 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.131707 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.131710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.131837 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.132211 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.132216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.132221 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.132599 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:17.132858 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:17.133212 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 14:55:17.133429 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 14:55:17 dist-test-slave-p3ml krb5kdc[23321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.60@KRBTEST.COM for kudu/127.0.241.60@KRBTEST.COM
Jun 19 14:55:17 dist-test-slave-p3ml krb5kdc[23321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.61@KRBTEST.COM
I20260619 14:55:17.145534 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.134310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53556)
0619 14:55:17.134461 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:17.134479 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:17.134573 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:17.134834 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:17.134838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:17.135177 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:17.135310 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:17.135343 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.136288 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.136291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:17.137091 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:17.137101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.137753 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.137758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:17.137988 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.138521 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:17.138536 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:17.140332 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:17.143196 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:17.143202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:17.143205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:17.143613 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:17.143943 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:17.143949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:17.143955 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:17.144080 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:17.144526 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:17.144530 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:17.144668 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 14:55:17.144833 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 14:55:17.145115 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260619 14:55:17.145583 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.134449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53556 (local address 127.0.241.60:37053)
0619 14:55:17.134596 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.134600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.134617 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:55:17.134653 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.134656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.134699 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.134823 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.135493 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.136172 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.137222 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.137613 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.137752 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.140503 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:17.140507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:17.140510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:17.140534 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:17.143057 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.143718 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.143725 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.143729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.143844 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.144173 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.144180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.144183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.144420 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:17.144558 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:17.144920 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 14:55:17.145258 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":187,"server-negotiator.queue_time_us":44}
I20260619 14:55:17.149991 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.135476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.0.241.61:44625)
0619 14:55:17.135606 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.135610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.136447 (+ 837us) server_negotiation.cc:407] Connection header received
0619 14:55:17.136611 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.136617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.136682 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.136780 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.137905 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.138776 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.140044 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.140443 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.140571 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.144466 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:17.144477 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:17.144480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:17.144507 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:17.147643 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.148250 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.148256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.148260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.148405 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.148738 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.148742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.148746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.149052 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:17.149312 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:17.149471 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 14:55:17.149686 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:55:17.151255 23549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2: Bootstrap starting.
I20260619 14:55:17.154570 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:17.155074 23549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:17.156131 23549 log.cc:824] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:17.158773 23549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2: No bootstrap required, opened a new log
I20260619 14:55:17.157186 23376 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60130
I20260619 14:55:17.165534 23549 raft_consensus.cc:348] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } }
I20260619 14:55:17.165843 23549 raft_consensus.cc:729] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d14f69d073f644efa72a52413a101ef2, State: Initialized, Role: FOLLOWER
I20260619 14:55:17.166285 23549 consensus_queue.cc:260] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } }
I20260619 14:55:17.167204 23556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } } }
I20260619 14:55:17.167619 23556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:17.168236 23549 sys_catalog.cc:564] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:55:17.178843 23567 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:55:17.179006 23567 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:55:17.176599 23482 master_runner.cc:186] Error getting master registration for 127.0.241.62:45685: OK, instance_id { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" instance_seqno: 1781880916054161 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:55:17.187886 23482 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:55:17.193064 23401 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } attempt: 4
Jun 19 14:55:17 dist-test-slave-p3ml krb5kdc[23321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for kudu/127.0.241.60@KRBTEST.COM
I20260619 14:55:17.208760 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.193803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.0.241.60:37053)
0619 14:55:17.194053 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.194059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.194083 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:55:17.194235 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.194240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.194308 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.194410 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.195354 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.196606 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.198408 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.199150 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.199266 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.203171 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:17.203177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:17.203181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:17.203218 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:17.206077 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.206720 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.206727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.206731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.206855 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.207224 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.207230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.207233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.207639 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:17.207814 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:17.208171 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 14:55:17.208378 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 14:55:17.208760 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.193747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53568)
0619 14:55:17.193956 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:17.193977 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:17.194105 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:17.194468 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:17.194474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:17.194991 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:17.195207 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:17.195221 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.196777 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.196784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:17.198246 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:17.198261 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.199302 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.199308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:17.199582 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.200177 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:17.200201 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:17.202987 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:17.206225 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:17.206231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:17.206236 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:17.206605 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:17.206966 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:17.206974 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:17.206977 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:17.207118 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:17.207755 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:17.207760 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:17.207921 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 14:55:17.208078 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 14:55:17.208378 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 14:55:17.214562 23401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3: Bootstrap starting.
I20260619 14:55:17.218626 23401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:17.219774 23401 log.cc:824] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:17.222471 23401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3: No bootstrap required, opened a new log
I20260619 14:55:17.228180 23401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } }
I20260619 14:55:17.228525 23401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b11fb47e4f449dfbed4e50c9abd56e3, State: Initialized, Role: FOLLOWER
I20260619 14:55:17.228994 23401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } }
I20260619 14:55:17.229945 23570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } } }
I20260619 14:55:17.230334 23570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:17.231077 23401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:17.231259 23376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 14:55:17.231535 23376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:17.235189 23376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } }
I20260619 14:55:17.238718 23524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d14f69d073f644efa72a52413a101ef2"
I20260619 14:55:17.239205 23524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:17.242640 23524 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b11fb47e4f449dfbed4e50c9abd56e3 in term 1.
I20260619 14:55:17.240747 23334 master_runner.cc:186] Error getting master registration for 127.0.241.61:44625: OK, instance_id { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" instance_seqno: 1781880916554127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:55:17.248914 23376 leader_election.cc:288] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [CANDIDATE]: Term 1 election: Requested vote from peers a49c15caeed94e7694e21033d42ae208 (127.0.241.61:44625), d14f69d073f644efa72a52413a101ef2 (127.0.241.60:37053)
I20260619 14:55:17.249006 23351 leader_election.cc:302] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b11fb47e4f449dfbed4e50c9abd56e3, d14f69d073f644efa72a52413a101ef2; no voters:
I20260619 14:55:17.249555 23570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:17.255522 23334 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 14:55:17.256560 23581 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:55:17.257098 23581 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:55:17.259951 23570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [term 1 LEADER]: Becoming Leader. State: Replica: 9b11fb47e4f449dfbed4e50c9abd56e3, State: Running, Role: LEADER
I20260619 14:55:17.261034 23570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } }
I20260619 14:55:17.272903 23583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b11fb47e4f449dfbed4e50c9abd56e3. Latest consensus state: current_term: 1 leader_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } } }
I20260619 14:55:17.273388 23583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:17.275467 23475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } attempt: 5
I20260619 14:55:17.284518 23589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 14:55:17 dist-test-slave-p3ml krb5kdc[23321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.61@KRBTEST.COM for kudu/127.0.241.60@KRBTEST.COM
I20260619 14:55:17.288671 23589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 14:55:17.293895 23355 tablet.cc:2369] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:55:17.296855 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.276332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:37053 (local address 127.0.0.1:53576)
0619 14:55:17.276779 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:17.276800 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:17.276940 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:17.277997 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:17.278006 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:17.278651 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:17.278916 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:17.278930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.280835 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.280843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:17.282612 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:17.282629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.284148 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.284155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:17.284397 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.285130 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:17.285158 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:17.288094 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:17.292541 (+ 4447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:17.292548 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:17.292552 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:17.293096 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:17.293556 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:17.293563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:17.293568 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:17.294800 (+ 1232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:17.295565 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:17.295573 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:17.295767 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 14:55:17.295936 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 14:55:17.296330 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289}
I20260619 14:55:17.298091 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.277204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53576 (local address 127.0.241.60:37053)
0619 14:55:17.277418 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.277426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.277445 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:55:17.277521 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.277529 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.277630 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.277754 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.279134 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.280634 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.282833 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.283765 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.283895 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.288366 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:17.288374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:17.288379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:17.288422 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:17.292368 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.293239 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.293246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.293250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.293419 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.294986 (+ 1567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.294994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.294997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.295339 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:17.295488 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:17.297297 (+ 1809us) server_negotiation.cc:299] Negotiation successful
0619 14:55:17.297594 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:55:17.304934 23475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208: Bootstrap starting.
I20260619 14:55:17.308969 23475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:17.310508 23475 log.cc:824] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:17.311381 23524 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [term 1 FOLLOWER]: Refusing update from remote peer 9b11fb47e4f449dfbed4e50c9abd56e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:17.312726 23583 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:55:17.317690 23503 tablet.cc:2369] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:55:17.328557 23475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208: No bootstrap required, opened a new log
I20260619 14:55:17.336077 23556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b11fb47e4f449dfbed4e50c9abd56e3. Latest consensus state: current_term: 1 leader_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } } }
I20260619 14:55:17.336438 23556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:17.353801 23593 mvcc.cc:205] Tried to move back new op lower bound from 7298584237293686784 to 7298584237110546432. Current Snapshot: MvccSnapshot[applied={T|T < 7298584237293686784}]
I20260619 14:55:17.364121 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.314715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.0.241.60:37053)
0619 14:55:17.317039 (+ 2324us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.317047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.332139 (+ 15092us) server_negotiation.cc:407] Connection header received
0619 14:55:17.335561 (+ 3422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.335568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.335718 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.335803 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.337937 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.340162 (+ 2225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.344284 (+ 4122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.344993 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.345093 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.351508 (+ 6415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:17.351514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:17.351518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:17.351558 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:17.356047 (+ 4489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.357917 (+ 1870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.357926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.357933 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.358300 (+ 367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.359265 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.359269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.359272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.360272 (+ 1000us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:17.360884 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:17.362400 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0619 14:55:17.363094 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2174}
I20260619 14:55:17.365605 23556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } } }
I20260619 14:55:17.365875 23556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:17.378206 23570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } } }
I20260619 14:55:17.378538 23570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:17.386831 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.318663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60146 (local address 127.0.241.62:45685)
0619 14:55:17.318836 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.318842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.332638 (+ 13796us) server_negotiation.cc:407] Connection header received
0619 14:55:17.334678 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.334686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.334769 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.334869 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.339543 (+ 4674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.340619 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.344045 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.344731 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.344836 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.372555 (+ 27719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:17.372563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:17.372568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:17.372612 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:17.376658 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.378951 (+ 2293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.378957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.378961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.379119 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.380281 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.380286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.380290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.380678 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:17.381003 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:17.386221 (+ 5218us) server_negotiation.cc:299] Negotiation successful
0619 14:55:17.386454 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:55:17.389379 23589 catalog_manager.cc:1347] Generated new cluster ID: e3b47f7cf4e247429e0556adc700c7a8
I20260619 14:55:17.389482 23589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:17.389443 23475 raft_consensus.cc:348] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } }
I20260619 14:55:17.390158 23475 raft_consensus.cc:729] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49c15caeed94e7694e21033d42ae208, State: Initialized, Role: FOLLOWER
I20260619 14:55:17.390966 23475 consensus_queue.cc:260] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } }
I20260619 14:55:17.404062 23599 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } } }
I20260619 14:55:17.404580 23599 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:17.406549 23450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a49c15caeed94e7694e21033d42ae208"
I20260619 14:55:17.407111 23449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:17.409678 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.325565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.0.241.61:44625)
0619 14:55:17.325785 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.325792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.343928 (+ 18136us) server_negotiation.cc:407] Connection header received
0619 14:55:17.346722 (+ 2794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.346729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.346818 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.346894 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.368654 (+ 21760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.369787 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.381304 (+ 11517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:17.382062 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:17.382210 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:17.402221 (+ 20011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:17.402229 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:17.402234 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:17.402284 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:17.406158 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.407101 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.407119 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.407123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.407287 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:17.407746 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:17.407751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:17.407754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:17.408157 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:17.408408 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:17.409018 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 14:55:17.409279 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 14:55:17.410897 23449 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [term 1 FOLLOWER]: Refusing update from remote peer 9b11fb47e4f449dfbed4e50c9abd56e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:17.411913 23583 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:17.414119 23450 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b11fb47e4f449dfbed4e50c9abd56e3 in term 1.
I20260619 14:55:17.420598 23589 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:17.427394 23589 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:17.435525 23475 sys_catalog.cc:564] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:17.454378 23589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3: Generated new TSK 0
I20260619 14:55:17.455415 23589 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:17.461449 23599 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b11fb47e4f449dfbed4e50c9abd56e3. Latest consensus state: current_term: 1 leader_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } } }
I20260619 14:55:17.461740 23599 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:17.461987 23603 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } } }
I20260619 14:55:17.462076 23570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b11fb47e4f449dfbed4e50c9abd56e3" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 45685 } } peers { permanent_uuid: "a49c15caeed94e7694e21033d42ae208" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 44625 } } peers { permanent_uuid: "d14f69d073f644efa72a52413a101ef2" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 37053 } } }
I20260619 14:55:17.462301 23603 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:17.462345 23570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b11fb47e4f449dfbed4e50c9abd56e3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:17.514011 23616 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 14:55:17.552456 23554 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:17.550445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36544 (local address 127.0.241.62:45685)
0619 14:55:17.550805 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.550811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.550830 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:55:17.550898 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.550903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.550992 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.551101 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.552196 (+ 1095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36544: BlockingRecv error: recv got EOF from 127.0.0.1:36544 (error 108)
Metrics: {"server-negotiator.queue_time_us":195}
W20260619 14:55:17.554881 23552 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:17.550764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.0.241.60:37053)
0619 14:55:17.550950 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.550956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.553240 (+ 2284us) server_negotiation.cc:407] Connection header received
0619 14:55:17.553523 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.553528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.553612 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.553680 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.554628 (+ 948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54094: BlockingRecv error: recv got EOF from 127.0.0.1:54094 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 14:55:17.558048 23479 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:17.555815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.0.241.61:44625)
0619 14:55:17.556004 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.556011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.556563 (+ 552us) server_negotiation.cc:407] Connection header received
0619 14:55:17.556793 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.556799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.556895 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.556967 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.557835 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43970: BlockingRecv error: recv got EOF from 127.0.0.1:43970 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 14:55:17.593876 23479 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:17.590630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.0.241.61:44625)
0619 14:55:17.590824 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.590830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.591989 (+ 1159us) server_negotiation.cc:407] Connection header received
0619 14:55:17.592312 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.592318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.592492 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.592570 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.593577 (+ 1007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43986: BlockingRecv error: recv got EOF from 127.0.0.1:43986 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 14:55:17.601553 23479 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:17.599395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43998 (local address 127.0.241.61:44625)
0619 14:55:17.599604 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.599609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.600038 (+ 429us) server_negotiation.cc:407] Connection header received
0619 14:55:17.600288 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.600293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.600372 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.600438 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.601336 (+ 898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43998: BlockingRecv error: recv got EOF from 127.0.0.1:43998 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260619 14:55:17.610855 23479 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:17.608512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.0.241.61:44625)
0619 14:55:17.608659 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.608665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.609109 (+ 444us) server_negotiation.cc:407] Connection header received
0619 14:55:17.609390 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.609396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.609524 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.609601 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.610646 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44002: BlockingRecv error: recv got EOF from 127.0.0.1:44002 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 14:55:17.620668 23479 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:17.618526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.0.241.61:44625)
0619 14:55:17.618720 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.618726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.619240 (+ 514us) server_negotiation.cc:407] Connection header received
0619 14:55:17.619424 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.619457 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.619559 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.619673 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.620465 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44016: BlockingRecv error: recv got EOF from 127.0.0.1:44016 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260619 14:55:17.635800 23479 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:17.633246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.0.241.61:44625)
0619 14:55:17.633463 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.633470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.634070 (+ 600us) server_negotiation.cc:407] Connection header received
0619 14:55:17.634340 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.634346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.634443 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.634558 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.635542 (+ 984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44032: BlockingRecv error: recv got EOF from 127.0.0.1:44032 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 14:55:17.657456 23479 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:17.655097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.0.241.61:44625)
0619 14:55:17.655290 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.655297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.655864 (+ 567us) server_negotiation.cc:407] Connection header received
0619 14:55:17.656076 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.656082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.656173 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.656287 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.657199 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44040: BlockingRecv error: recv got EOF from 127.0.0.1:44040 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 14:55:17.699208 23479 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:17.696738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.0.241.61:44625)
0619 14:55:17.696929 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.696935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.697523 (+ 588us) server_negotiation.cc:407] Connection header received
0619 14:55:17.697739 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.697745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.697856 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.697980 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.698934 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44056: BlockingRecv error: recv got EOF from 127.0.0.1:44056 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 14:55:17.768877 23479 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:17.766481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.0.241.61:44625)
0619 14:55:17.766770 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.766779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.767107 (+ 328us) server_negotiation.cc:407] Connection header received
0619 14:55:17.767465 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.767470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.767571 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.767698 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.768647 (+ 949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44064: BlockingRecv error: recv got EOF from 127.0.0.1:44064 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260619 14:55:17.911054 23479 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:17.909102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.0.241.61:44625)
0619 14:55:17.909357 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:17.909364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:17.909613 (+ 249us) server_negotiation.cc:407] Connection header received
0619 14:55:17.909843 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:17.909848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:17.909945 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:17.910066 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:17.910802 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44068: BlockingRecv error: recv got EOF from 127.0.0.1:44068 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260619 14:55:18.173043 23479 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:18.171101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.0.241.61:44625)
0619 14:55:18.171357 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:18.171364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:18.171643 (+ 279us) server_negotiation.cc:407] Connection header received
0619 14:55:18.171862 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:18.171867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:18.171933 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:18.172040 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:18.172813 (+ 773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44080: BlockingRecv error: recv got EOF from 127.0.0.1:44080 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:55:18.180895 23567 catalog_manager.cc:1259] Loaded cluster ID: e3b47f7cf4e247429e0556adc700c7a8
I20260619 14:55:18.180985 23567 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2: loading cluster ID for follower catalog manager: success
I20260619 14:55:18.183378 23567 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2: acquiring CA information for follower catalog manager: success
I20260619 14:55:18.185513 23567 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d14f69d073f644efa72a52413a101ef2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:55:18.518342 23616 catalog_manager.cc:1259] Loaded cluster ID: e3b47f7cf4e247429e0556adc700c7a8
I20260619 14:55:18.518491 23616 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208: loading cluster ID for follower catalog manager: success
I20260619 14:55:18.521096 23616 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208: acquiring CA information for follower catalog manager: success
I20260619 14:55:18.522966 23616 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a49c15caeed94e7694e21033d42ae208: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:55:18.701505 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:18.694747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.0.241.61:44625)
0619 14:55:18.695553 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:18.695557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:18.695577 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:55:18.695647 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:18.695651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:18.695791 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:18.695902 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:18.696469 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:18.697221 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:18.699144 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:18.699593 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:18.699697 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:18.700281 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:18.700812 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:18.700923 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:18.701076 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 14:55:18.701151 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":481,"threads_started":1}
I20260619 14:55:18.714529 23686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:18.708091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.0.241.60:37053)
0619 14:55:18.708759 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:18.708767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:18.708788 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:55:18.708891 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:18.708895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:18.709032 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:18.709198 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:18.709591 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:18.710568 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:18.712137 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:18.712592 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:18.712747 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:18.713277 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:18.713786 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:18.713975 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:18.714119 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 14:55:18.714206 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":424,"threads_started":1}
I20260619 14:55:18.744299 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:18.721925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.0.241.62:45685)
0619 14:55:18.730221 (+ 8296us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:18.730231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:18.735723 (+ 5492us) server_negotiation.cc:407] Connection header received
0619 14:55:18.736338 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:18.736351 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:18.736636 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:18.736845 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:18.737963 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:18.739138 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:18.741332 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:18.741913 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:18.742072 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:18.742742 (+ 670us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:18.743201 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:18.743290 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:18.743823 (+ 533us) server_negotiation.cc:299] Negotiation successful
0619 14:55:18.743919 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8125,"thread_start_us":8013,"threads_started":1}
I20260619 14:55:18.744948 23686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:18.722326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.0.241.60:37053)
0619 14:55:18.722507 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:18.722515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:18.737603 (+ 15088us) server_negotiation.cc:407] Connection header received
0619 14:55:18.738120 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:18.738127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:18.738216 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:18.738300 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:18.739333 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:18.740493 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:18.742448 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:18.743101 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:18.743207 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:18.744048 (+ 841us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:18.744193 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:18.744262 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:18.744550 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 14:55:18.744637 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:55:18.745846 23680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:18.721371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.0.241.61:44625)
0619 14:55:18.721556 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:18.721562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:18.737916 (+ 16354us) server_negotiation.cc:407] Connection header received
0619 14:55:18.738188 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:18.738195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:18.738281 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:18.738341 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:18.739361 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:18.740440 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:18.743607 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:18.744226 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:18.744333 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:18.744999 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:18.745156 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:18.745220 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:18.745455 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 14:55:18.745535 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:55:18.751749 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 23334
I20260619 14:55:18.752799 23396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:55:18.862969 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 23334
I20260619 14:55:18.884114 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 23408
I20260619 14:55:18.885238 23470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:55:18.994551 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 23408
I20260619 14:55:19.014402 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 23482
I20260619 14:55:19.015568 23544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:55:19.120352 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 23482
2026-06-19T14:55:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4834 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:19 dist-test-slave-p3ml krb5kdc[23700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:19 dist-test-slave-p3ml krb5kdc[23700](info): set up 2 sockets
Jun 19 14:55:19 dist-test-slave-p3ml krb5kdc[23700](info): commencing operation
krb5kdc: starting...
W20260619 14:55:20.228040 964 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/test-user.keytab.
Jun 19 14:55:20 dist-test-slave-p3ml krb5kdc[23700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.241.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
Entry for principal kudu/127.0.241.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab.
I20260619 14:55:20.480871 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35465
--webserver_interface=127.0.241.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:41399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:35465,127.0.241.61:34163,127.0.241.60:42531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:20.811044 23713 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:55:20.811379 23713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:20.811545 23713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:20.811681 23713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:20.818888 23713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:20.818971 23713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:20.819075 23713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:20.819165 23713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:20.819288 23713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:20.829366 23713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:41399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:35465,127.0.241.61:34163,127.0.241.60:42531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kudu_127.0.241.62.keytab
--principal=kudu/127.0.241.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.62:35465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.241.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:20.830772 23713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:20.832239 23713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:20.844192 23719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:20.844221 23721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:20.844247 23718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:20.845428 23713 server_base.cc:1034] running on GCE node
I20260619 14:55:20.846154 23713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:20.847395 23713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:20.848728 23713 hybrid_clock.cc:648] HybridClock initialized: now 1781880920848641 us; error 49 us; skew 500 ppm
I20260619 14:55:20.849107 23713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:20 dist-test-slave-p3ml krb5kdc[23700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:20.852367 23713 init.cc:377] Logged in from keytab as kudu/127.0.241.62@KRBTEST.COM (short username kudu)
I20260619 14:55:20.853807 23713 webserver.cc:460] Webserver started at http://127.0.241.62:45239/ using document root <none> and password file <none>
I20260619 14:55:20.854458 23713 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:20.854571 23713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:20.854902 23713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:20.857882 23713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/data/instance:
uuid: "1f45383cedf2402c82ec7efdc5ecee54"
format_stamp: "Formatted at 2026-06-19 14:55:20 on dist-test-slave-p3ml"
server_key: "c64d216c8462f58d6272377a7b2b1071"
server_key_iv: "75e3884994a4fac6ad5152ee737b7bbd"
server_key_version: "encryptionkey@0"
I20260619 14:55:20.858675 23713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/wal/instance:
uuid: "1f45383cedf2402c82ec7efdc5ecee54"
format_stamp: "Formatted at 2026-06-19 14:55:20 on dist-test-slave-p3ml"
server_key: "c64d216c8462f58d6272377a7b2b1071"
server_key_iv: "75e3884994a4fac6ad5152ee737b7bbd"
server_key_version: "encryptionkey@0"
I20260619 14:55:20.863741 23713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 14:55:20.866977 23728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:20.868259 23713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:20.868438 23713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/wal
uuid: "1f45383cedf2402c82ec7efdc5ecee54"
format_stamp: "Formatted at 2026-06-19 14:55:20 on dist-test-slave-p3ml"
server_key: "c64d216c8462f58d6272377a7b2b1071"
server_key_iv: "75e3884994a4fac6ad5152ee737b7bbd"
server_key_version: "encryptionkey@0"
I20260619 14:55:20.868634 23713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:20.889598 23713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:20.893992 23713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:20.894263 23713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:20.912871 23713 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.62:35465
I20260619 14:55:20.912890 23779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.62:35465 every 8 connection(s)
I20260619 14:55:20.914177 23713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:20.917798 23780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:20.919927 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 23713
I20260619 14:55:20.920125 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-0/wal/instance
I20260619 14:55:20.920697 964 external_mini_cluster.cc:1428] Setting key ec670b46ae48dfa748581d5051013a5b
I20260619 14:55:20.926957 23780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:20 dist-test-slave-p3ml krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.241.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.61@KRBTEST.COM" created.
I20260619 14:55:20.949385 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:20.932030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35114 (local address 127.0.241.62:35465)
0619 14:55:20.932828 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:20.932835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:20.932895 (+ 60us) server_negotiation.cc:407] Connection header received
0619 14:55:20.933996 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:20.934001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:20.934227 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:20.934428 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:20.935712 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:20.936887 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:20.938220 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:20.938749 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:20.938941 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:20.942403 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:20.942410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:20.942420 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:20.942446 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:20.945202 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:20.945926 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:20.945935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:20.945939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:20.946156 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:20.946544 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:20.946548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:20.946551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:20.947130 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:20.947276 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:20.947866 (+ 590us) server_negotiation.cc:299] Negotiation successful
0619 14:55:20.948180 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":598,"thread_start_us":417,"threads_started":1}
I20260619 14:55:20.949420 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:20.930607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35465 (local address 127.0.0.1:35114)
0619 14:55:20.931422 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:20.931519 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:20.933651 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:20.934449 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:20.934455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:20.935118 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:20.935507 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:20.935527 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:20.937038 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:20.937046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:20.938084 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:20.938096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:20.938978 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:20.938985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:20.939174 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:20.939839 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:20.939860 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:20.942191 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:20.945346 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:20.945358 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:20.945368 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:20.945737 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:20.946266 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:20.946269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:20.946271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:20.946410 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:20.947257 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:20.947263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:20.947427 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 14:55:20.947752 (+ 325us) client_negotiation.cc:240] Negotiation successful
0619 14:55:20.948337 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":134,"thread_start_us":359,"threads_started":1}
I20260619 14:55:20.952618 23780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:55:20.953704 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:20.953356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36262)
0619 14:55:20.953511 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:20.953608 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 14:55:20.953966 23729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
I20260619 14:55:20.955533 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:20.955126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36266)
0619 14:55:20.955285 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:20.955352 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:55:20.955749 23780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:34163: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kudu_127.0.241.61.keytab.
Entry for principal kudu/127.0.241.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kudu_127.0.241.61.keytab.
I20260619 14:55:20.976915 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.61:34163
--webserver_interface=127.0.241.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kudu_127.0.241.61.keytab
--principal=kudu/127.0.241.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:41399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:35465,127.0.241.61:34163,127.0.241.60:42531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:55:21.005049 23780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } attempt: 1
I20260619 14:55:21.006278 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.005743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36280)
0619 14:55:21.005978 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.006103 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 14:55:21.007386 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.006938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36288)
0619 14:55:21.007107 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.007205 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:55:21.007620 23780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:34163: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
I20260619 14:55:21.075851 23780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } attempt: 2
I20260619 14:55:21.076995 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.076483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36294)
0619 14:55:21.076708 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.076820 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 14:55:21.078045 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.077641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36296)
0619 14:55:21.077784 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.077882 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 14:55:21.078272 23780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:34163: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
I20260619 14:55:21.169569 23780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } attempt: 3
I20260619 14:55:21.170806 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.170269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36306)
0619 14:55:21.170519 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.170631 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 14:55:21.171913 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.171522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36314)
0619 14:55:21.171669 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.171753 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 14:55:21.172106 23780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:34163: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
W20260619 14:55:21.296994 23787 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:55:21.297305 23787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:21.297374 23787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:21.297449 23787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:21.304296 23787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:21.304370 23787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:21.304421 23787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:21.304466 23787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:21.304519 23787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:21.314327 23787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:41399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:35465,127.0.241.61:34163,127.0.241.60:42531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kudu_127.0.241.61.keytab
--principal=kudu/127.0.241.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.61:34163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.241.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:21.315357 23780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } attempt: 4
I20260619 14:55:21.315595 23787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:21.316708 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.316101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36330)
0619 14:55:21.316441 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.316544 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260619 14:55:21.317152 23787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:55:21.317906 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.317451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36346)
0619 14:55:21.317636 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.317749 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 14:55:21.318193 23780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.61:34163: Network error: Client connection negotiation failed: client connection to 127.0.241.61:34163: connect: Connection refused (error 111)
W20260619 14:55:21.329008 23792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:21.328977 23795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:21.328933 23793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:21.329895 23787 server_base.cc:1034] running on GCE node
I20260619 14:55:21.330626 23787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:21.331941 23787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:21.333156 23787 hybrid_clock.cc:648] HybridClock initialized: now 1781880921333100 us; error 24 us; skew 500 ppm
I20260619 14:55:21.333417 23787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:21 dist-test-slave-p3ml krb5kdc[23700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:21.341255 23787 init.cc:377] Logged in from keytab as kudu/127.0.241.61@KRBTEST.COM (short username kudu)
I20260619 14:55:21.342725 23787 webserver.cc:460] Webserver started at http://127.0.241.61:37449/ using document root <none> and password file <none>
I20260619 14:55:21.343420 23787 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:21.343590 23787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:21.343920 23787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:21.346804 23787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/data/instance:
uuid: "545005d144264c44acdf37aaac8398d7"
format_stamp: "Formatted at 2026-06-19 14:55:21 on dist-test-slave-p3ml"
server_key: "7146ca2d681768547d42ee13091378e7"
server_key_iv: "5f9984d038722306ef688d53edeeef39"
server_key_version: "encryptionkey@0"
I20260619 14:55:21.347640 23787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/wal/instance:
uuid: "545005d144264c44acdf37aaac8398d7"
format_stamp: "Formatted at 2026-06-19 14:55:21 on dist-test-slave-p3ml"
server_key: "7146ca2d681768547d42ee13091378e7"
server_key_iv: "5f9984d038722306ef688d53edeeef39"
server_key_version: "encryptionkey@0"
I20260619 14:55:21.352547 23787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260619 14:55:21.355748 23802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:21.356981 23787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:55:21.357167 23787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/wal
uuid: "545005d144264c44acdf37aaac8398d7"
format_stamp: "Formatted at 2026-06-19 14:55:21 on dist-test-slave-p3ml"
server_key: "7146ca2d681768547d42ee13091378e7"
server_key_iv: "5f9984d038722306ef688d53edeeef39"
server_key_version: "encryptionkey@0"
I20260619 14:55:21.357370 23787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:21.399959 23787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:21.403839 23787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:21.404111 23787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:21.422096 23787 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.61:34163
I20260619 14:55:21.422116 23853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.61:34163 every 8 connection(s)
I20260619 14:55:21.423388 23787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/data/info.pb
I20260619 14:55:21.427157 23854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:21.429921 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 23787
I20260619 14:55:21.430124 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-1/wal/instance
I20260619 14:55:21.430549 964 external_mini_cluster.cc:1428] Setting key 5b6ce007423d427e5768c439233952cd
I20260619 14:55:21.434175 23854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:21 dist-test-slave-p3ml krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.61@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.241.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.241.60@KRBTEST.COM" created.
I20260619 14:55:21.456164 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.436923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.0.241.62:35465)
0619 14:55:21.437142 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:21.437147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:21.438619 (+ 1472us) server_negotiation.cc:407] Connection header received
0619 14:55:21.440913 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:21.440922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:21.441018 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:21.441133 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:21.443168 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.444341 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.445810 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.446564 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.446728 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.450200 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:21.450207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:21.450210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:21.450244 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:21.453025 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.453836 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.453844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.453847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.453972 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.454378 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.454384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.454387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.454720 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:21.454885 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:21.455582 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 14:55:21.455817 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 14:55:21.457108 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.437481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35465 (local address 127.0.0.1:35120)
0619 14:55:21.438376 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.438443 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:21.440629 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:21.441472 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:21.441483 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:21.442396 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:21.442983 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:21.443000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.444490 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.444494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:21.445649 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:21.445660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.446819 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.446824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:21.447022 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.447800 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:21.447823 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:21.449961 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:21.453184 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:21.453193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:21.453207 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:21.453704 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:21.454092 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:21.454098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:21.454102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:21.454266 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:21.454848 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:21.454857 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:21.455178 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 14:55:21.455474 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 14:55:21.455979 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":357,"threads_started":1}
I20260619 14:55:21.458827 23854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:21 dist-test-slave-p3ml krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.61@KRBTEST.COM for kudu/127.0.241.61@KRBTEST.COM
I20260619 14:55:21.475664 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.459662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36352)
0619 14:55:21.459940 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.459959 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:21.460058 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:21.461105 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:21.461112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:21.461570 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:21.461782 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:21.461795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.463800 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.463807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:21.464822 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:21.464833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.466025 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.466029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:21.466149 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.466732 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:21.466750 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:21.469006 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:21.473274 (+ 4268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:21.473278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:21.473281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:21.473657 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:21.474050 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:21.474053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:21.474055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:21.474187 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:21.474907 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:21.474910 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:21.475015 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 14:55:21.475120 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 14:55:21.475320 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260619 14:55:21.476183 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.459713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.0.241.61:34163)
0619 14:55:21.460593 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:21.460602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:21.460631 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:55:21.460695 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:21.460705 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:21.460951 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:21.461090 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:21.461960 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.463665 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.464969 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.465888 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.466028 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.469226 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:21.469241 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:21.469259 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:21.469304 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:21.473123 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.473787 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.473798 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.473804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.473945 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.474295 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.474302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.474308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.474775 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:21.474980 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:21.475328 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 14:55:21.475643 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":534,"threads_started":1}
I20260619 14:55:21.478461 23854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:55:21.479624 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.479130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40472)
0619 14:55:21.479313 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.479422 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 14:55:21.479848 23804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
I20260619 14:55:21.481343 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.481037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40474)
0619 14:55:21.481182 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.481253 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 14:55:21.481559 23854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:42531: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.241.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kudu_127.0.241.60.keytab.
Entry for principal kudu/127.0.241.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kudu_127.0.241.60.keytab.
I20260619 14:55:21.490835 964 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskydHOLK/build/asan/bin/kudu
/tmp/dist-test-taskydHOLK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.60:42531
--webserver_interface=127.0.241.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kudu_127.0.241.60.keytab
--principal=kudu/127.0.241.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.241.20:41399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.241.62:35465,127.0.241.61:34163,127.0.241.60:42531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:55:21.530937 23854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } attempt: 1
I20260619 14:55:21.532052 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.531618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40482)
0619 14:55:21.531808 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.531913 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 14:55:21.533017 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.532707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40494)
0619 14:55:21.532862 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.532925 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:55:21.533203 23854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:42531: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
I20260619 14:55:21.601402 23854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } attempt: 2
I20260619 14:55:21.602473 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.602042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40506)
0619 14:55:21.602231 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.602325 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 14:55:21.603511 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.603158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40522)
0619 14:55:21.603294 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.603351 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 14:55:21.603709 23854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:42531: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
I20260619 14:55:21.617432 23780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } attempt: 5
Jun 19 14:55:21 dist-test-slave-p3ml krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for kudu/127.0.241.61@KRBTEST.COM
I20260619 14:55:21.631582 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.618173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.0.241.61:34163)
0619 14:55:21.618328 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:21.618332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:21.618356 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:55:21.618573 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:21.618577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:21.618657 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:21.618739 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:21.619880 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.620555 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.622451 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.622840 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.622925 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.626375 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:21.626380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:21.626383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:21.626410 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:21.629130 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.629739 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.629747 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.629750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.629873 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.630240 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.630246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.630249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.630516 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:21.630675 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:21.631037 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 14:55:21.631247 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:55:21.631594 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.618077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36354)
0619 14:55:21.618279 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.618300 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:21.618439 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:21.618823 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:21.618829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:21.619470 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:21.619724 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:21.619750 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.620692 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.620698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:21.622302 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:21.622317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.622997 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.623003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:21.623187 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.623852 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:21.623874 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:21.626191 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:21.629252 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:21.629259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:21.629263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:21.629622 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:21.629980 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:21.629985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:21.629989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:21.630128 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:21.630632 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:21.630637 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:21.630796 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 14:55:21.630942 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 14:55:21.631218 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 14:55:21.632539 23780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:55:21.633608 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.633157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40534)
0619 14:55:21.633334 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.633426 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 14:55:21.634665 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.634226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40536)
0619 14:55:21.634399 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.634516 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:55:21.634917 23780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:42531: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
I20260619 14:55:21.686137 23780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } attempt: 1
I20260619 14:55:21.687485 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.686830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40548)
0619 14:55:21.687080 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.687215 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 14:55:21.688546 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.688176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40562)
0619 14:55:21.688331 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.688418 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 14:55:21.688776 23780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:42531: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
I20260619 14:55:21.694893 23854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } attempt: 3
I20260619 14:55:21.695835 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.695377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40570)
0619 14:55:21.695601 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.695688 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 14:55:21.696843 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.696466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40582)
0619 14:55:21.696682 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.696747 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260619 14:55:21.697044 23854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:42531: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
I20260619 14:55:21.757006 23780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } attempt: 2
I20260619 14:55:21.758285 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.757719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40592)
0619 14:55:21.757958 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.758081 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 14:55:21.759373 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.758988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40594)
0619 14:55:21.759146 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.759233 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 14:55:21.759676 23780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:42531: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
W20260619 14:55:21.820556 23861 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:55:21.820866 23861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:21.820932 23861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:21.821007 23861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:21.828013 23861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:21.828100 23861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:21.828153 23861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:21.828207 23861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:21.828279 23861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:21.837692 23861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.241.20:41399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.241.62:35465,127.0.241.61:34163,127.0.241.60:42531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/krb5kdc/kudu_127.0.241.60.keytab
--principal=kudu/127.0.241.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.241.60:42531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.241.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 13:58:13 UTC on ccycloud.quasar-psecfj.root.comops.site
build id 4455767
ASAN enabled
I20260619 14:55:21.838871 23861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:21.840260 23861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:55:21.840247 23854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } attempt: 4
I20260619 14:55:21.841334 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.840867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40604)
0619 14:55:21.841097 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.841182 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 14:55:21.842511 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.842193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40618)
0619 14:55:21.842340 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.842413 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 14:55:21.842749 23854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:42531: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
W20260619 14:55:21.852212 23867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:21.852236 23866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:21.852164 23869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:21.853013 23861 server_base.cc:1034] running on GCE node
I20260619 14:55:21.853775 23861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:21.855099 23861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:21.856348 23861 hybrid_clock.cc:648] HybridClock initialized: now 1781880921856282 us; error 34 us; skew 500 ppm
I20260619 14:55:21.856681 23861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 14:55:21 dist-test-slave-p3ml krb5kdc[23700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:21.859838 23861 init.cc:377] Logged in from keytab as kudu/127.0.241.60@KRBTEST.COM (short username kudu)
I20260619 14:55:21.861223 23861 webserver.cc:460] Webserver started at http://127.0.241.60:43899/ using document root <none> and password file <none>
I20260619 14:55:21.861976 23861 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:21.862063 23861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:21.862396 23861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:21.865355 23861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/data/instance:
uuid: "de30642051b74db4a711b41ab9a5b229"
format_stamp: "Formatted at 2026-06-19 14:55:21 on dist-test-slave-p3ml"
server_key: "44d566aacefe7d5327aba354fe8ed39e"
server_key_iv: "c816fd44cef1ebad09f99904345f2f49"
server_key_version: "encryptionkey@0"
I20260619 14:55:21.865911 23780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } attempt: 3
I20260619 14:55:21.866191 23861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/wal/instance:
uuid: "de30642051b74db4a711b41ab9a5b229"
format_stamp: "Formatted at 2026-06-19 14:55:21 on dist-test-slave-p3ml"
server_key: "44d566aacefe7d5327aba354fe8ed39e"
server_key_iv: "c816fd44cef1ebad09f99904345f2f49"
server_key_version: "encryptionkey@0"
I20260619 14:55:21.866964 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.866519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40622)
0619 14:55:21.866728 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.866808 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 14:55:21.868022 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.867656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40628)
0619 14:55:21.867847 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.867920 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 14:55:21.868258 23780 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.241.60:42531: Network error: Client connection negotiation failed: client connection to 127.0.241.60:42531: connect: Connection refused (error 111)
I20260619 14:55:21.871651 23861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 14:55:21.875020 23876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:21.876360 23861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:21.876545 23861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/data,/tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/wal
uuid: "de30642051b74db4a711b41ab9a5b229"
format_stamp: "Formatted at 2026-06-19 14:55:21 on dist-test-slave-p3ml"
server_key: "44d566aacefe7d5327aba354fe8ed39e"
server_key_iv: "c816fd44cef1ebad09f99904345f2f49"
server_key_version: "encryptionkey@0"
I20260619 14:55:21.876770 23861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:21.888818 23861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:21.892814 23861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:21.893100 23861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:21.910872 23861 rpc_server.cc:307] RPC server started. Bound to: 127.0.241.60:42531
I20260619 14:55:21.910893 23927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.241.60:42531 every 8 connection(s)
I20260619 14:55:21.912235 23861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/data/info.pb
I20260619 14:55:21.913609 964 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskydHOLK/build/asan/bin/kudu as pid 23861
I20260619 14:55:21.913803 964 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskydHOLK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880565277815-964-0/minicluster-data/master-2/wal/instance
I20260619 14:55:21.914285 964 external_mini_cluster.cc:1428] Setting key 6eff4c80e4d457790d81897ed4a4f9b4
I20260619 14:55:21.917165 23928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:21.928103 23928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:21 dist-test-slave-p3ml krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.60@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
Jun 19 14:55:21 dist-test-slave-p3ml krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.60@KRBTEST.COM
I20260619 14:55:21.945911 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.931226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.0.241.62:35465)
0619 14:55:21.931485 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:21.931493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:21.931935 (+ 442us) server_negotiation.cc:407] Connection header received
0619 14:55:21.932362 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:21.932367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:21.932446 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:21.932557 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:21.933797 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.934483 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.936030 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.936448 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.936580 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.940277 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:21.940283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:21.940295 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:21.940324 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:21.942902 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.943722 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.943727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.943730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.943859 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.944295 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.944299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.944303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.944557 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:21.944707 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:21.945350 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 14:55:21.945563 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 14:55:21.946977 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.931096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.62:35465 (local address 127.0.0.1:35134)
0619 14:55:21.931791 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.931822 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:21.932139 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:21.932615 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:21.932622 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:21.933248 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:21.933631 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:21.933648 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.934646 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.934660 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:21.935894 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:21.935906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.936684 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.936691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:21.937016 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.937747 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:21.937780 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:21.940097 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:21.943042 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:21.943055 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:21.943066 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:21.943594 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:21.943975 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:21.943980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:21.943983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:21.944169 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:21.944692 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:21.944699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:21.944993 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 14:55:21.945226 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 14:55:21.945712 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":61,"thread_start_us":333,"threads_started":1}
I20260619 14:55:21.946995 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.916872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40644 (local address 127.0.241.60:42531)
0619 14:55:21.929011 (+ 12139us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:21.929030 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:21.929672 (+ 642us) server_negotiation.cc:407] Connection header received
0619 14:55:21.932126 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:21.932152 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:21.932401 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:21.932616 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:21.934075 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.935241 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.936557 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.937147 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.937308 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.941365 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:21.941373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:21.941383 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:21.941415 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:21.944248 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.944865 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.944871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.944873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.944994 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.945394 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.945398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.945400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.945714 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:21.946020 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:21.946331 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 14:55:21.946574 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":11859,"thread_start_us":12427,"threads_started":1}
I20260619 14:55:21.948859 23928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:21 dist-test-slave-p3ml krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.60@KRBTEST.COM for kudu/127.0.241.61@KRBTEST.COM
Jun 19 14:55:21 dist-test-slave-p3ml krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.62@KRBTEST.COM
I20260619 14:55:21.963181 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.949613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.61:34163 (local address 127.0.0.1:36368)
0619 14:55:21.949787 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.949805 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:21.949925 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:21.950358 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:21.950365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:21.950833 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:21.951015 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:21.951030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.952105 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.952109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:21.953153 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:21.953167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.953856 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.953863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:21.954078 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.954655 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:21.954677 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:21.956941 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:21.960517 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:21.960521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:21.960524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:21.961039 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:21.961481 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:21.961484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:21.961486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:21.961664 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:21.962290 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:21.962293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:21.962416 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 14:55:21.962567 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 14:55:21.962815 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 14:55:21.963230 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.949629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.0.241.61:34163)
0619 14:55:21.949818 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:21.949825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:21.949847 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:55:21.950078 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:21.950084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:21.950203 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:21.950304 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:21.951175 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.951969 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.953299 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.953715 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.953805 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.957126 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:21.957131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:21.957134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:21.957164 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:21.960351 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.961162 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.961168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.961173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.961351 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.961780 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.961786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.961789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.962147 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:21.962326 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:21.962659 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 14:55:21.962866 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 14:55:21.964447 23928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:55:21.965211 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.950117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.0.241.62:35465)
0619 14:55:21.950301 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:21.950306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:21.950370 (+ 64us) server_negotiation.cc:407] Connection header received
0619 14:55:21.950537 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:21.950541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:21.950588 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:21.950659 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:21.951943 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.952727 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.954198 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.954571 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.954655 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.959068 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:21.959074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:21.959078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:21.959117 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:21.962455 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.963041 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.963046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.963050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.963188 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.963739 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.963743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.963745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.964085 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:21.964403 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:21.964688 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 14:55:21.964857 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 14:55:21 dist-test-slave-p3ml krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.60@KRBTEST.COM for kudu/127.0.241.60@KRBTEST.COM
Jun 19 14:55:21 dist-test-slave-p3ml krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.241.61@KRBTEST.COM
I20260619 14:55:21.977238 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.965732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.0.241.60:42531)
0619 14:55:21.965875 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:21.965878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:21.965892 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:55:21.965926 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:21.965929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:21.965974 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:21.966089 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:21.966946 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.967670 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.968731 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.969143 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.969273 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.972045 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:21.972050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:21.972053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:21.972077 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:21.974839 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.975447 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.975454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.975460 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.975577 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.975937 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.975944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.975947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.976200 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:21.976342 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:21.976671 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 14:55:21.976893 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:55:21.977334 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.965154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40650)
0619 14:55:21.965349 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.965370 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:21.965454 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:21.966174 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:21.966181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:21.966632 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:21.966796 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:21.966806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.967783 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.967790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:21.968603 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:21.968613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.969276 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.969282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:21.969457 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.969978 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:21.969993 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:21.971873 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:21.974962 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:21.974969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:21.974973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:21.975320 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:21.975688 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:21.975694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:21.975697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:21.975836 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:21.976309 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:21.976313 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:21.976445 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 14:55:21.976600 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 14:55:21.976992 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":128}
I20260619 14:55:21.981060 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.966811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.0.241.61:34163)
0619 14:55:21.966959 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:21.966966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:21.966985 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:55:21.967149 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:21.967154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:21.967222 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:21.967322 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:21.968456 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.969175 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.970486 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:21.970898 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:21.971037 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:21.975537 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:21.975541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:21.975544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:21.975572 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:21.978538 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.979163 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.979169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.979172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.979313 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:21.979720 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:21.979726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:21.979729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:21.979973 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:21.980273 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:21.980543 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 14:55:21.980763 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:55:21.982311 23928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229: Bootstrap starting.
I20260619 14:55:21.985572 964 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:21.986009 23928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:21.987097 23928 log.cc:824] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:21.990265 23928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229: No bootstrap required, opened a new log
I20260619 14:55:21.988036 23755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1f45383cedf2402c82ec7efdc5ecee54"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35144
I20260619 14:55:22.000864 23928 raft_consensus.cc:348] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } }
I20260619 14:55:22.001255 23928 raft_consensus.cc:729] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de30642051b74db4a711b41ab9a5b229, State: Initialized, Role: FOLLOWER
I20260619 14:55:22.001749 23928 consensus_queue.cc:260] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } }
I20260619 14:55:22.002674 23934 sys_catalog.cc:455] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } } }
I20260619 14:55:22.002996 23934 sys_catalog.cc:458] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:22.003939 23928 sys_catalog.cc:564] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:55:22.013159 23945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:55:22.013290 23945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:55:22.011215 23861 master_runner.cc:186] Error getting master registration for 127.0.241.62:35465: OK, instance_id { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" instance_seqno: 1781880920907377 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:55:22.022253 23861 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:55:22.045475 23780 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } attempt: 4
Jun 19 14:55:22 dist-test-slave-p3ml krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.62@KRBTEST.COM for kudu/127.0.241.60@KRBTEST.COM
I20260619 14:55:22.063470 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.047293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.0.241.60:42531)
0619 14:55:22.047520 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.047528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.047555 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:55:22.047623 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.047628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.047727 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.047829 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.048919 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.050058 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.051986 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.052789 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.052943 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.056749 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:22.056757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:22.056762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:22.056808 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:22.060197 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:22.061069 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:22.061074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:22.061078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:22.061225 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:22.061707 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:22.061712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:22.061715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:22.062174 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:22.062341 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.062816 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.063047 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:55:22.063510 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.046303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40656)
0619 14:55:22.046549 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:22.046572 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:22.046715 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:22.047894 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:22.047902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:22.048503 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:22.048742 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:22.048755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.050225 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.050231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:22.051834 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:22.051852 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.052957 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.052965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:22.053224 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.053947 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:22.053967 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:22.056549 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:22.060349 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:22.060355 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:22.060359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:22.060924 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:22.061361 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:22.061367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:22.061370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:22.061577 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:22.062303 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:22.062310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:22.062519 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 14:55:22.062692 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 14:55:22.063011 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 14:55:22.069955 23780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54: Bootstrap starting.
I20260619 14:55:22.073596 23780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:22.074681 23780 log.cc:824] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:22.077538 23780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54: No bootstrap required, opened a new log
I20260619 14:55:22.083146 23780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } }
I20260619 14:55:22.083513 23780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f45383cedf2402c82ec7efdc5ecee54, State: Initialized, Role: FOLLOWER
I20260619 14:55:22.083976 23780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } }
I20260619 14:55:22.084988 23948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } } }
I20260619 14:55:22.085400 23948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:22.086110 23780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:22.086251 23755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 14:55:22.086483 23755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:22.089916 23755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } }
I20260619 14:55:22.092114 23755 leader_election.cc:288] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [CANDIDATE]: Term 1 election: Requested vote from peers 545005d144264c44acdf37aaac8398d7 (127.0.241.61:34163), de30642051b74db4a711b41ab9a5b229 (127.0.241.60:42531)
I20260619 14:55:22.092393 23903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f45383cedf2402c82ec7efdc5ecee54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "de30642051b74db4a711b41ab9a5b229"
I20260619 14:55:22.092859 23903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:22.096345 23903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f45383cedf2402c82ec7efdc5ecee54 in term 1.
I20260619 14:55:22.096946 23730 leader_election.cc:302] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f45383cedf2402c82ec7efdc5ecee54, de30642051b74db4a711b41ab9a5b229; no voters:
I20260619 14:55:22.097380 23948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:22.107455 23948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [term 1 LEADER]: Becoming Leader. State: Replica: 1f45383cedf2402c82ec7efdc5ecee54, State: Running, Role: LEADER
I20260619 14:55:22.108003 23948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } }
I20260619 14:55:22.114547 23958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f45383cedf2402c82ec7efdc5ecee54. Latest consensus state: current_term: 1 leader_uuid: "1f45383cedf2402c82ec7efdc5ecee54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } } }
I20260619 14:55:22.114957 23958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:22.118995 23961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:22.134770 23961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:22.141948 23854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } attempt: 5
W20260619 14:55:22.145087 23882 tablet.cc:2369] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:55:22.147025 23734 tablet.cc:2369] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:55:22 dist-test-slave-p3ml krb5kdc[23700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.241.61@KRBTEST.COM for kudu/127.0.241.60@KRBTEST.COM
I20260619 14:55:22.169175 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.142475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.241.60:42531 (local address 127.0.0.1:40668)
0619 14:55:22.142856 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:22.142877 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:22.143017 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:22.144057 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:22.144062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:22.144573 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:22.144815 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:22.144829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.146370 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.146378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:22.147794 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:22.147808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.149106 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.149112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:22.149346 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.150145 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:22.150211 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:22.155333 (+ 5122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:22.162550 (+ 7217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:22.162557 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:22.162562 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:22.163472 (+ 910us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:22.164296 (+ 824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:22.164303 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:22.164307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:22.164597 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:22.165784 (+ 1187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:22.165790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:22.166063 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 14:55:22.166306 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 14:55:22.167064 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260619 14:55:22.170651 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.143359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.0.241.60:42531)
0619 14:55:22.143570 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.143579 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.143603 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:55:22.143679 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.143684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.143766 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.143862 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.145027 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.146201 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.147967 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.148739 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.148862 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.155698 (+ 6836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:22.155708 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:22.155716 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:22.155789 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:22.162283 (+ 6494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:22.163703 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:22.163767 (+ 64us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:22.163771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:22.164089 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:22.164900 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:22.164908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:22.164911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:22.165578 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:22.166837 (+ 1259us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.167397 (+ 560us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.168436 (+ 1039us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 14:55:22.152522 23713 master_runner.cc:186] Error getting master registration for 127.0.241.61:34163: OK, instance_id { permanent_uuid: "545005d144264c44acdf37aaac8398d7" instance_seqno: 1781880921416922 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:55:22.171345 23713 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:55:22.178580 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.139188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35160 (local address 127.0.241.62:35465)
0619 14:55:22.139363 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.139369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.143954 (+ 4585us) server_negotiation.cc:407] Connection header received
0619 14:55:22.144230 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.144238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.144313 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.144384 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.159551 (+ 15167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.160670 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.163142 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.164216 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.164339 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.168631 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:22.168640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:22.168647 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:22.168695 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:22.173493 (+ 4798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:22.176154 (+ 2661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:22.176159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:22.176162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:22.176326 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:22.176815 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:22.176820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:22.176823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:22.177164 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:22.177431 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.177995 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.178229 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:55:22.182734 23854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7: Bootstrap starting.
I20260619 14:55:22.186720 23854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:22.188222 23854 log.cc:824] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:22.193595 23903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [term 1 FOLLOWER]: Refusing update from remote peer 1f45383cedf2402c82ec7efdc5ecee54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:22.194561 23958 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:22.211556 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.139695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.0.241.61:34163)
0619 14:55:22.139938 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.139949 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.159447 (+ 19498us) server_negotiation.cc:407] Connection header received
0619 14:55:22.159664 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.159671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.159751 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.159870 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.175544 (+ 15674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.176638 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.180191 (+ 3553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.180828 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.180931 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.196296 (+ 15365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:22.196305 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:22.196310 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:22.196354 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:22.200105 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:22.202593 (+ 2488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:22.202599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:22.202603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:22.202753 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:22.204006 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:22.204011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:22.204015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:22.204475 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:22.204723 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.210901 (+ 6178us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.211157 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 14:55:22.219873 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.160172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.0.241.60:42531)
0619 14:55:22.171324 (+ 11152us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.171330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.171387 (+ 57us) server_negotiation.cc:407] Connection header received
0619 14:55:22.171481 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.171486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.171622 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.171847 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.179387 (+ 7540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.180340 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.185017 (+ 4677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.185859 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.186022 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.206083 (+ 20061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:22.206090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:22.206095 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:22.206135 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:22.210216 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:22.214395 (+ 4179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:22.214413 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:22.214417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:22.214567 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:22.215322 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:22.215339 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:22.215343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:22.215766 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:22.218643 (+ 2877us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.218774 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.219018 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10952,"thread_start_us":15536,"threads_started":1}
I20260619 14:55:22.220424 23854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7: No bootstrap required, opened a new log
I20260619 14:55:22.220878 23934 sys_catalog.cc:455] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f45383cedf2402c82ec7efdc5ecee54. Latest consensus state: current_term: 1 leader_uuid: "1f45383cedf2402c82ec7efdc5ecee54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } } }
I20260619 14:55:22.221192 23934 sys_catalog.cc:458] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:22.231552 23854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } }
I20260619 14:55:22.231938 23854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545005d144264c44acdf37aaac8398d7, State: Initialized, Role: FOLLOWER
I20260619 14:55:22.232437 23854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } }
I20260619 14:55:22.239362 23977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } } }
I20260619 14:55:22.239785 23977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:22.240008 23948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f45383cedf2402c82ec7efdc5ecee54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } } }
I20260619 14:55:22.240350 23948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:22.240705 23970 mvcc.cc:205] Tried to move back new op lower bound from 7298584257288110080 to 7298584256959856640. Current Snapshot: MvccSnapshot[applied={T|T < 7298584257288110080}]
I20260619 14:55:22.240962 23974 mvcc.cc:205] Tried to move back new op lower bound from 7298584257288110080 to 7298584256959856640. Current Snapshot: MvccSnapshot[applied={T|T < 7298584257288110080}]
I20260619 14:55:22.241591 23854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:22.241672 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1f45383cedf2402c82ec7efdc5ecee54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "545005d144264c44acdf37aaac8398d7"
I20260619 14:55:22.242120 23829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:22.242631 23961 catalog_manager.cc:1347] Generated new cluster ID: e35346583a32407bab1f50d65df4bed6
I20260619 14:55:22.242751 23961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:22.246740 23829 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f45383cedf2402c82ec7efdc5ecee54 in term 1.
I20260619 14:55:22.248471 23828 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [term 1 FOLLOWER]: Refusing update from remote peer 1f45383cedf2402c82ec7efdc5ecee54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:22.250468 23958 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:22.257414 23934 sys_catalog.cc:455] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f45383cedf2402c82ec7efdc5ecee54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } } }
I20260619 14:55:22.257745 23934 sys_catalog.cc:458] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:22.277653 23961 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:22.282724 23961 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:22.282759 23977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f45383cedf2402c82ec7efdc5ecee54. Latest consensus state: current_term: 1 leader_uuid: "1f45383cedf2402c82ec7efdc5ecee54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } } }
I20260619 14:55:22.282802 23986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1f45383cedf2402c82ec7efdc5ecee54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } } }
I20260619 14:55:22.283136 23977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:22.283139 23986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:22.284399 23948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1f45383cedf2402c82ec7efdc5ecee54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f45383cedf2402c82ec7efdc5ecee54" member_type: VOTER last_known_addr { host: "127.0.241.62" port: 35465 } } peers { permanent_uuid: "545005d144264c44acdf37aaac8398d7" member_type: VOTER last_known_addr { host: "127.0.241.61" port: 34163 } } peers { permanent_uuid: "de30642051b74db4a711b41ab9a5b229" member_type: VOTER last_known_addr { host: "127.0.241.60" port: 42531 } } }
I20260619 14:55:22.284701 23948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:22.300799 23961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f45383cedf2402c82ec7efdc5ecee54: Generated new TSK 0
I20260619 14:55:22.302197 23961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:22.363121 23995 catalog_manager.cc:1259] Loaded cluster ID: e35346583a32407bab1f50d65df4bed6
I20260619 14:55:22.363337 23995 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7: loading cluster ID for follower catalog manager: success
I20260619 14:55:22.367326 23995 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7: acquiring CA information for follower catalog manager: success
I20260619 14:55:22.370049 23995 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 545005d144264c44acdf37aaac8398d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 14:55:22.463094 23784 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:22.441826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.0.241.62:35465)
0619 14:55:22.442071 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.442077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.460176 (+ 18099us) server_negotiation.cc:407] Connection header received
0619 14:55:22.460527 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.460536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.460690 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.460815 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.462737 (+ 1922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35176: BlockingRecv error: recv got EOF from 127.0.0.1:35176 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 14:55:22.469594 23858 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:22.448918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.0.241.61:34163)
0619 14:55:22.449141 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.449147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.463756 (+ 14609us) server_negotiation.cc:407] Connection header received
0619 14:55:22.467792 (+ 4036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.467802 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.467932 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.468023 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.469267 (+ 1244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36386: BlockingRecv error: recv got EOF from 127.0.0.1:36386 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 14:55:22.473412 23931 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:22.470653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.0.241.60:42531)
0619 14:55:22.470850 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.470855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.471499 (+ 644us) server_negotiation.cc:407] Connection header received
0619 14:55:22.471792 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.471798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.471875 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.471953 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.473119 (+ 1166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40698: BlockingRecv error: recv got EOF from 127.0.0.1:40698 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:55:22.523377 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.504064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.0.241.61:34163)
0619 14:55:22.504274 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.504281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.514862 (+ 10581us) server_negotiation.cc:407] Connection header received
0619 14:55:22.515480 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.515488 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.515720 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.515890 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:22.516669 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.517886 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.520084 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.520846 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.520959 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.521801 (+ 842us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:22.522465 (+ 664us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:22.522599 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.522865 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.522985 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 14:55:22.570559 23931 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:22.557284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.0.241.60:42531)
0619 14:55:22.557517 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.557525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.568208 (+ 10683us) server_negotiation.cc:407] Connection header received
0619 14:55:22.568804 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.568812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.568958 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.569046 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.570232 (+ 1186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40710: BlockingRecv error: recv got EOF from 127.0.0.1:40710 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:55:22.625175 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.606830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.0.241.61:34163)
0619 14:55:22.607053 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.607060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.617665 (+ 10605us) server_negotiation.cc:407] Connection header received
0619 14:55:22.618210 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.618218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.618351 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.618429 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:22.619244 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.620526 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.622720 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.623425 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.623563 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.624225 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:22.624404 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:22.624470 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.624746 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.624840 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 14:55:22.673678 23931 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:22.661929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40716 (local address 127.0.241.60:42531)
0619 14:55:22.662122 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.662130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.671893 (+ 9763us) server_negotiation.cc:407] Connection header received
0619 14:55:22.672166 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.672173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.672296 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.672431 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.673437 (+ 1006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40716: BlockingRecv error: recv got EOF from 127.0.0.1:40716 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:55:22.687135 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.679740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.0.241.61:34163)
0619 14:55:22.679939 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.679946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.680309 (+ 363us) server_negotiation.cc:407] Connection header received
0619 14:55:22.680558 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.680565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.680671 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.680783 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:22.681258 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.682470 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.684547 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.685338 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.685487 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.686027 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:22.686187 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:22.686298 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.686472 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.686573 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 14:55:22.697460 23931 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:22.694641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.0.241.60:42531)
0619 14:55:22.694846 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.694853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.695962 (+ 1109us) server_negotiation.cc:407] Connection header received
0619 14:55:22.696151 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.696157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.696250 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.696353 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.697212 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40720: BlockingRecv error: recv got EOF from 127.0.0.1:40720 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 14:55:22.716506 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.709341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.0.241.61:34163)
0619 14:55:22.709537 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.709544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.710108 (+ 564us) server_negotiation.cc:407] Connection header received
0619 14:55:22.710311 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.710317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.710407 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.710496 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:22.711025 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.712246 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.714315 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.715005 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.715156 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.715710 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:22.715866 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:22.715968 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.716147 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.716231 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 14:55:22.726181 23931 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:22.723741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.0.241.60:42531)
0619 14:55:22.723997 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.724004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.724552 (+ 548us) server_negotiation.cc:407] Connection header received
0619 14:55:22.724739 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.724746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.724847 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.724962 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.725933 (+ 971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40732: BlockingRecv error: recv got EOF from 127.0.0.1:40732 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
I20260619 14:55:22.748485 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.741411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.0.241.61:34163)
0619 14:55:22.741625 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.741631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.742179 (+ 548us) server_negotiation.cc:407] Connection header received
0619 14:55:22.742407 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.742414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.742514 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.742611 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:22.743093 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.744250 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.746217 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.746929 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.747088 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.747575 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:22.747743 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:22.747852 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.748014 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.748104 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 14:55:22.758158 23931 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:22.756240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.0.241.60:42531)
0619 14:55:22.756448 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.756454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.756738 (+ 284us) server_negotiation.cc:407] Connection header received
0619 14:55:22.756885 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.756891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.756990 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.757109 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.757839 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40746: BlockingRecv error: recv got EOF from 127.0.0.1:40746 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:55:22.783493 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.777362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.0.241.61:34163)
0619 14:55:22.777606 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.777614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.777814 (+ 200us) server_negotiation.cc:407] Connection header received
0619 14:55:22.778021 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.778027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.778132 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.778251 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:22.778631 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.779945 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.781377 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.782106 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.782254 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.782643 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:22.782793 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:22.782887 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.783010 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.783139 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260619 14:55:22.792040 23931 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:22.789993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.0.241.60:42531)
0619 14:55:22.790193 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.790200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.790534 (+ 334us) server_negotiation.cc:407] Connection header received
0619 14:55:22.790746 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.790754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.790858 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.790977 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.791784 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40756: BlockingRecv error: recv got EOF from 127.0.0.1:40756 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:55:22.834919 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.828677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36446 (local address 127.0.241.61:34163)
0619 14:55:22.828882 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.828891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.829244 (+ 353us) server_negotiation.cc:407] Connection header received
0619 14:55:22.829458 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.829464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.829654 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.829796 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:22.830185 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.831422 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.832794 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.833542 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.833689 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.834023 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:22.834184 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:22.834301 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.834392 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.834484 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 14:55:22.844050 23931 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:22.841589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.0.241.60:42531)
0619 14:55:22.841812 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.841820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.842664 (+ 844us) server_negotiation.cc:407] Connection header received
0619 14:55:22.842787 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.842794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.842920 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.843053 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.843800 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40766: BlockingRecv error: recv got EOF from 127.0.0.1:40766 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 14:55:22.938429 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.911477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.0.241.61:34163)
0619 14:55:22.912958 (+ 1481us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.912968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.932123 (+ 19155us) server_negotiation.cc:407] Connection header received
0619 14:55:22.932361 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.932367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.932453 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.932532 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:22.933081 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.934242 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.936299 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:22.936966 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:22.937072 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:22.937680 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:22.937806 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:22.937861 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:22.938057 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 14:55:22.938135 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 14:55:22.948226 23931 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:22.946048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.0.241.60:42531)
0619 14:55:22.946223 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:22.946229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:22.946739 (+ 510us) server_negotiation.cc:407] Connection header received
0619 14:55:22.946903 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:22.946910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:22.947016 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:22.947135 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:22.948004 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40776: BlockingRecv error: recv got EOF from 127.0.0.1:40776 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:55:23.015192 23945 catalog_manager.cc:1259] Loaded cluster ID: e35346583a32407bab1f50d65df4bed6
I20260619 14:55:23.015293 23945 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229: loading cluster ID for follower catalog manager: success
I20260619 14:55:23.017760 23945 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229: acquiring CA information for follower catalog manager: success
I20260619 14:55:23.019383 23945 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de30642051b74db4a711b41ab9a5b229: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:55:23.084144 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:23.079542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.0.241.61:34163)
0619 14:55:23.079752 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:23.079761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:23.080193 (+ 432us) server_negotiation.cc:407] Connection header received
0619 14:55:23.080363 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:23.080367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:23.080452 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:23.080537 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:23.080906 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:23.081579 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:23.082815 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:23.083198 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:23.083284 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:23.083620 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:23.083725 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:23.083789 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:23.083889 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 14:55:23.083947 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:55:23.095149 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:23.089677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40784 (local address 127.0.241.60:42531)
0619 14:55:23.089885 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:23.089889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:23.090203 (+ 314us) server_negotiation.cc:407] Connection header received
0619 14:55:23.090381 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:23.090385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:23.090542 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:23.090643 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:23.091022 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:23.091824 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:23.093201 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:23.093603 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:23.093694 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:23.094118 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:23.094663 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:23.094769 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:23.094863 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 14:55:23.094930 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 14:55:23.110332 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:23.101195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.0.241.61:34163)
0619 14:55:23.101347 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:23.101351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:23.104099 (+ 2748us) server_negotiation.cc:407] Connection header received
0619 14:55:23.104367 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:23.104373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:23.104474 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:23.104557 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:23.105653 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:23.106723 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:23.108461 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:23.108911 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:23.109041 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:23.109533 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:23.109656 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:23.109717 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:23.109951 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 14:55:23.110041 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:55:23.110917 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:23.104209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.0.241.60:42531)
0619 14:55:23.104378 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:23.104383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:23.104926 (+ 543us) server_negotiation.cc:407] Connection header received
0619 14:55:23.105200 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:23.105205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:23.105293 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:23.105390 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:23.105910 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:23.106682 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:23.109160 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:23.109561 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:23.109629 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:23.110204 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:23.110322 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:23.110414 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:23.110593 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 14:55:23.110667 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:55:23.111888 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:23.100572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.0.241.62:35465)
0619 14:55:23.103986 (+ 3414us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:23.103992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:23.104009 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:55:23.104125 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:23.104133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:23.104381 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:23.104893 (+ 512us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:23.105538 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:23.106668 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:23.109156 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:23.109810 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:23.109936 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:23.110461 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:23.110919 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:23.111425 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:23.111528 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 14:55:23.111594 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3295,"thread_start_us":3194,"threads_started":1}
I20260619 14:55:23.116365 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 23713
I20260619 14:55:23.117312 23775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:55:23.225797 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 23713
I20260619 14:55:23.246593 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 23787
I20260619 14:55:23.247730 23849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:55:23.355532 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 23787
I20260619 14:55:23.375334 964 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskydHOLK/build/asan/bin/kudu pid 23861
I20260619 14:55:23.376477 23923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 14:55:23.484123 964 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskydHOLK/build/asan/bin/kudu with pid 23861
2026-06-19T14:55:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4364 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9199 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (358168 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST